Explore insights on real-time edge computing, AI, and cloud solutions. Learn how our technology optimizes performance across industries.

Blog

If you think about making your own twitter-ish app, you will soon stumble upon a few super interesting challenges.

A twitter style app will allow anyone to sign up, tweet, and then to follow people and see their tweets.

The feature Signing Up requires some kind of database to store the users. If this is just for your closest friends or a few thousand tweeters, SQLite will be fine.

Of course, you also need some back-end to handle the sign-ups and talk with the database. You could use a LAMP stack, Ruby on Rails or roller-skates, or Node.js or whatever. It doesn't really matter. Unless you want your app to be scalable, then you may want to make the back-end a stateless micro-service that talks with a stateful database like MariaDB, Postgres or MongoDB - each of which can be scaled up if you put some effort into it.

Once signed up, your users must have a way to become followers. In other words; they need a way to discover other users. That means that our app must have some search functionality. So - when someone signs up they could search for their friends, or people with similar interest (or opposite interests for the evangelists). What about other local users? How cool would it be to be able to find and follow people in your own village or neighborhood? Or in the same coffee shop or nightclub?

Since all internet connected devices have some kind of location awareness (so that advertisers can feed us the most relevant ads), how about exploiting that to locate nearby people? Getting your location from your device is trivial. But a simple database like SQLite has no geo-location awareness. So we need a database that knows about geo-locations to implement that feature. A proper geo-location aware database should allow us to directly query for the n closest entities of any geo-location (latitude, longitude) in the world.

Another way to find followers is to see who else is following the people we follow. That can be implemented using a traditional SQL database - to a certain extent. If we want to walk several levels into the follower network we are really in the realm of a graph database. A graph database tracks relations between objects, and allow us to make simple and fast queries to traverse the graph to fetch whatever objects we desire.

But the most important feature of them all is to make sure all of our followers get our tweets immediately. Not like Google Chat or Skype, that may take days to push a message notification. When we have something on our tongue, we want people to get it immediately so that they can react to it (read: Like it!).

The traditional way to implement a thing like this, using web technology, is polling. The client polls every second to see if any of the people it follows has sent any new messages. That's fine on a residential Internet connection with a PC, but not so much on a phone, where you pay for the traffic and where traffic drains the battery - not to mention the back-end resources. If you make your twitter for your friends, you just need one micro-service instance with Node.js, 5 with Ruby or 42 with Java, to serve 20 friends. On the other hand, if your app goes viral you have a real challenge. One million users, each polling each second, will be rather expensive to handle as you will need thousands of micro-services. A better approach is to push the messages to the device. That can be accomplished with grpc, push notifications, or just plain old web-sockets.

In a micro-service architecture, one way to handle pushing messages in real time to the clients is a pub-sub service. Think of it as queues or persistent streams, where each user has a personal queue or a stream she subscribes to. When someone publishes a message, the back-end copies the message to the queue or stream for each of her subscribers. As soon as the message is pushed to a queue or stream, the pub-sub service checks if the subscriber is connected, and immediately sends the message to those who are. Those who are off-line will get the messages as soon as they connect to their queue or stream.

So for our twitterish app, we need a trivial database for the users, a geo-location database, a graph database, and some kind of pub-sub service for the notifications. That's 4 stateful technologies that our stateless micro-service has to deal with (and we have to read up on, comprehend, deploy and interact with from the micro-service).

If the app goes viral, we also have to scale it from tens to thousands to millions of users. That's hard, especially if our users are spread all around the world. If we deploy our gang of 4 in Central Europe, users from USA, India, China, Australia will get latency and connectivity issues. (Even Netflix and HBO are unable to serve a global user base in a timely manner. I often switch off my TV before either of them starts to play the movie I want to watch). If your app is slow, you lose users. It's that simple.

So what if we could just make up some imaginary fairy technology that our stateless micro-service could connect to that would have low latency, scale to any size, and deal with the gang of 4 stateful technologies we require - from anywhere in the world? Then we could deploy our micro-services in Australia, in Asia, in Europe in USA, in UK - anywhere large clusters of users subscribed - and provide all of them with wonderful, reliable, low latency access to the latest fake news and trolling on the net? (Let's be realistic - if your app goes viral, all of the bad people will be there, but so will all of the good people, and we are the majority!)

Well - good news. The "fairy" technology is here!

I have spent the last few months working on it, after joining a small team of dedicated, talented people in India, Europe and USA. When it went online with an early free preview a few weeks ago, I wanted to spend a weekend or two making an app that consumed all of our main features - the gang of 4 - to get an idea about how easy or difficult it would be to actually use our service as a software architect and developer. So I made a simple app in Qt/QML that I called 

 (I'm more a dog person then a bird person) using our document database (a basic key/value store where the values are Json documents) for users and messages (barks), the graph database (edges attached to Json documents) for relations (followers), the Geo-location index type (for the Json documents) for Geo-location, and streams over websockets to receive new messages in real time.

It took me 46 hours and 39 minutes to do this. About 1/3 of the time was spent on figuring out how to implement all the client functionality in QML (usually I use C++ for all the logic - but I wanted to learn something new - and if someone wants to use my code as an example, JavaScript is much simpler to understand for most people than C++), 1/3 was spent working around bugs, figuring out undocumented features and tuning queries (don't worry - our documentation team and developers got a steady stream of feedback - so all of that is/will be fixed), and 1/3 was spent on implementing the thing.

I ended up with a very simple stateless micro-service back-end in Node.js dealing with subscriptions to the service, login, distributing barks (what Twitter refers to as tweets) and a few other things.

I could have distributed the barks in the client, but if the client is shut off or loses connectivity in the process, some subscribers may not see the barks - which will be devastating for the barker, so it was better architecture to put it in the back-end .

For me this was an expedition into unknown technological territory. I have not used geo-locations or graph databases in the past, and I have never designed a stateful, distributed application that's supposed to scale to any size just by deploying more back-end resources.

I did not make it secure, as I would have done with a real app. But that was a trade off for simplicity - I wanted the client to connect directly to the database API for as many things as possible, and only use the micro-service where it is really needed. That of course means that the client's user credentials need access to the actual databases, and this blows away pretty much all potential security. For example, if the user can write to the locations database to update her own location, she can also (by modifying the app or call the API with curl) change anyone else's location.

The Macrometa Fast Data platform is designed to offload the burden of scaling stateful components (the databases and the streams) from the application architecture and 'just work'. The service allows you to enable local endpoints all around the world just by clicking on a map in a dashboard. Then you can use geo DNS to allow your users to connect directly to the nearest endpoint of your database, or deploy micro-services in the same or close-by data-centers to handle the local users. Your document collections and streams are instantly replicated to all the geographic regions you enable. For a software architect or developer - using the database or the streams from a mobile app, webapp or micro-service is just as simple as connecting to Postgres, MongoDB or Redis from your code. You don't have to worry about deploying or scaling those components. It's just a nearby REST API endpoint (or web-sockets for the streams) with low latency and extra, cool features like geo-location and graph queries. I was totally blown away by this technology when I first learned about it a few months back. For a software architect or app developer - it opens up a totally new dimension of opportunities. Just take my barker app as an example - in one week, I created an app that can scale to millions of users all over the word with just a few clicks in the Macrometa dashboard - without doing anything at all in my architecture or code to deal with that.

Thank you for following me this far. If you think this was a cool read, please share it in the social media of your choice - even twitter.

Now - to prove that I am in fact a developer, and not some marketing guy or business person, I'll quote my update in today's Scrum call.

Boss: Did you start on the Big, Boring, Exhausting task I assigned to you?

Sign up for a free Macrometa account today by clicking 

https://github.com/Macrometacorp/barker-demo

How To Make Your Own Twitter In 5 Days

 to provide low latency and a fast and real time user experience. These kinds of games are hard to build because you need to understand low level networking inside game engines (like unity or unreal), and deal with lots of complexity when it comes to sending and receiving state data as well as storing it on a server for persistence. Now what if I told you that you could do all of this in a few hours without needing to deal with anything except a few API functions callable via JavaScript. Afterward, you'd have an 

 game with real time-data streaming and data storage.

In this tutorial, I'm going to show you how to use Macrometa to create a multiplayer, side scrolling platform game that uses Macrometa's streams to send messages to each player about the game state, controls (up, down, left, right and jump), and the database for storing persistent game state (thing like who is on what level, how many points collected, how many treasures like diamonds and keys collected etc).

It's probably good to play the game for few minutes to better understand the functionalities. The game is a single page app served off a CDN. You can start and play this game directly using below link and you can invite your friends to join by sharing the link with them):

 originally published by pubnub. The pubnub example tutorial can be found 

If you want real-time interaction among players, pubnub or socket.io won't work because both of their pub/sub architectures are centralized. That means that sending any data from the game client to each other will have to transit all the way to their central servers before being received by the other game clients. That just adds latency. Additionally - neither pubnub or socket.io let you persist game state so if you want to collect player stats in real-time - well you're going to have to use a database of your own or a cloud database - but guess what - those are centralized too and have high levels of latency to access.

While there are a lot of tools out there for writing games, most of them create more problems when you start handling real-time multiplayer interactions & state over tens and hundred of locations globally.

This is where something like Macrometa comes in. It provides ready made APIs, enabling you and your teams to build, deploy and run cross-region or globally – in real-time.

As Macrometa's Chief Architect, I am obviously biased. :-) So don’t take my word as fact. The point of this post is for you to try things yourself to determine what's easy or not. So let’s build a global, real-time, online multiplayer game called CastleKnight together!

After that, you can decide for yourself if Macrometa saves you and your team lots of time to focus on developing your game.

This game is a collaborative puzzle game that encourages you to work with your friends to collect the keys in clever ways.

Following are some useful capabilities we want incorporate into this game :

 i.e., reliable real-time game state and events and inputs across all connected players.

 i.e., incorporate interactive social features like chat across all connected players utilizing the streams feature of Macrometa platform.

 i.e., real-time score updates and game statistics to dashboards

 i.e., various levels in the game and player stats in each of those levels.

Obviously one can build more functionalities in the game, but I think for this post the above is sufficient. You are welcome to add more functionality.

The rest of the post is all about how we build the Game GUI and leverage 

 in the game. As a game developer, this is the part you should actually be focusing your precious time and efforts on.

Obviously you'll need a Macrometa account for this tutorial, but don't worry, it's free to try. Go ahead and sign up 

After signup, you will receive a tenant account along with credentials. After login, you should see a dashboard something like the image below.

You can get the source code for the game 

 to change it. Go ahead and clone the repository to a directory on your local machine.

After cloning, you should see something like this in your directory:

Step3: Configure the game to use your account

Open js/Config.js file and edit the following parameters to reflect your account:

CastleKnight is a single page app. So to run the game locally, you have to launch your local web server. If you have Mac OS or Linux (or have Python installed), open up your Terminal Application in your game folder and type in:

. There are some great tutorials out there on how to setup XAMPP on your machine.

Once you have your web server up and running, go to http://localhost:8000/ on your machine. Then navigate to the index.html file in your web browser and click on the link. You should now see a blank screen.

If you want to deploy and run the game from S3 then do the following. Go outside the current working directory i.e., tutorial-castleknight-game in this case. If you are using the AWS cli then run following command to recursively copy all files and folders inside the tutorial-castleknight-game folder to the S3 bucket.

Now go to the Properties tab in the AWS console for this bucket and open Static website hosting option. Then select the option Use this bucket to host a website and provide index.html for both Index document and Error document text fields. Click on save and the game is now live!

To begin, open up your main.js file. Following code fragment reads your credentials from Config.js and opens the connection to Macrometa data platform. Also it creates a geo-replicated document collection called occupancy to record the player occupancy in each level.

The following code fragment in loadingState.js loads the assets into the scene. We created an object called window.LoadingState with all of the loading state information inside of it. In the init() function, we made the sprite objects in the game look smoother by using the Phaser API this.game.renderer.renderSession.roundPixels = true;

In the preload function, we load the JSON level information from the data folder. This information is used to generate the various levels of the game. Then every asset that we will use in the game needs to be preloaded into cache.

Lastly we run the create() function that starts the game and loads whatever the window.globalCurrentLevel is.

Open again your main.js file. The following code fragment creates necessary global variables for the game.

Open chatEngine.js. The following code fragment initializes the chat engine and creates a geo-replicated stream between players to publish and receive chat messages in realtime around the world. The last part of the code adds the chat panel to the game GUI at the bottom.

This post became a lot longer than I originally anticipated, so I'll skip the code explanation for display of player interactions and elements in the game. Savvy javascript developers should be able to understand by looking at the code directly.

I hope you enjoyed this tutorial and it helps with your gaming project. If you notice, most of the code is about the game GUI i.e., display of assets, player interactions, etc. This frees the game developer to focus on what matters most. We’re looking forward to launching more multiplayer gaming tutorials in the future so stay tuned.

how to build your very own crypto arbitrage bot

CastleKnight - Using Macrometa's Edge Database & Streaming Engine To Build Low Latency, Realtime Multiplayer Online Games *With Bonus In-Game Chat

The emergence and the fast growth of Mobile Apps, Web Apps, Single Page Apps and APIs within the past few years is a telltale sign of the growing importance and demand for speed and faster user experiences by users. And this is not simply a psychological need for speed in our ever accelerating and connected world. It is a requirement.

Faster apps lead to better user engagement.

Faster apps lead to better user retention.

Despite the ever-accelerating pace of life, our reaction times remain constant, regardless of the type of application (online or offline), or medium (laptop, desktop, or mobile device).

 0–100 ms Instant 100–300 ms Small perceptible delay 300–1000 ms Machine is working 1,000+ ms Likely mental context switch 10,000+ ms Task is abandoned

For an application to feel instant, a perceptible response to user input must be provided within 10s of milliseconds. After a 1/2 second or more, the user’s flow and engagement with the initiated task is broken, and after 3 seconds have passed, unless progress feedback is provided, the task is frequently abandoned.

There are a number of technical solutions for improving an application’s performance, such as the data transfer capabilities of servers or routers, including compression and streamlining application roundtrip responses. But there is one thing that cannot be addressed with technology, and that’s proximity.

Solving Latency with Geography, Not Technology

Given the above, the most practical solution is to move the data geographically closer to the customer or end points for a business. This is the concept behind 

 (CDN) and caching sites. Most get deployed on the edge to provide faster access to common, or popular data and they bypass the ‘speed of transport network’ issues by getting popular content to the edge, where the customers are.

But CDN's were created to solve the problems of the "old web" - a web that was static and relied heavily on server-side rendering. They don't provide much for SaaS providers. SaaS applications are powered by query driven, dynamic data - and CDNs don't do well with dynamic data.

Macrometa brings the CDN into the cloud-native era by providing a global multi-model database and geo-replicated streams at the CDN edge. It means your apps always run on the edge with nothing to cache, purge or expire. The edge is the dynamic origin for every request and your users are always served the freshest state from the closest edge PoP.

Simply put, we may not be able to make the packets travel faster, but we can reduce the distance your dynamic data has to travel. This empowers you to serve your customers globally at high speed by enabling your Apps & APIs to utilize dynamic data in close proximity to your customers.

Given the benefits associated with making state (i.e., dynamic data) available in close geographical proximity to users, why don’t we see developers doing this? It is because maintaining state (i.e., dynamic data) at the edge across several locations is a very hard problem especially when one needs various consistency guarantees on that data.

It takes years and a team of engineers with deep expertise in distributed systems & databases to build this type of fast data platform. If you have the time, money, energy and skills then you can certainly build a multi-modal fast data platform like Macrometa. The industry can certainly benefit from more data platforms like Macrometa’s fast data platform that are coordination-free and utilize both consensus and convergence mechanisms.

On other hand, if your primary focus is in building Apps and APIs for your business needs. then the Macrometa platform can definitely help you serve your customers globally, at high speeds - thus fairly quickly, by providing a geo-distributed fast data platform with various consistency guarantees and other functionalities.

Let’s create the backend for an Address Book app in the 

. An app that is available globally at high speed (i.e., local read-write latencies) to its users by leveraging Macrometa’s fast data platform.

 in the above paragraph because typically doing the same exercise without using Macrometa’s platform will take months to build it (assuming the team has necessary expertise). Don’t take my word though. Verify it for yourself.

You need an account with Macrometa’s Fast Data platform to build the backend for the app. If needed, you can get a free developer account 

Faster apps lead to better user engagement, better user retention and higher conversions. The most practical solution is to move the data and application geographically closer to the customer or end points for your business.

Macrometa’s fast data platform enables dynamic data to be used like CDN by providing a global multi-model realtime database and geo-replicated streams at the edge. It means the backend for your apps always run on the edge with nothing to cache, purge or expire. The edge is the dynamic origin for every request from your app and your users are always served the freshest state from the closest edge PoP.

I hope the above shows how quick and easy it is to create a geo-distributed backend engine using Macrometa for your app to serve your users faster globally.

There are quite few other capabilities available in Macrometa’s Fast Data platform. Give it a try with a 

Serving Your Users Globally, Faster

Build Geo Distributed Apps with Macrometa

Here's a few examples of how Macrometa turns complex 

Build a multi region database that runs in 6 different data-centers across the world in about 40 seconds (the video is not trimmed in any way):

Create a data collection in the global database (like a table in an RDBMS), insert some data in it, and see it replicate with consistency guarantees in all the regions your fabric runs in:

Create a query to retrieve data from the collection, save and export the query as an API end point that you can call from anywhere in the world. The request will always be served from the closest datacenter to the requester

Call your restQL API end point from a web application such as a Single Page Application (SPA) or Progressive Web App (PWA): using the showContacts example from above:

Realtime data is easy with Macrometa - all changes to the database are automatically pushed to all clients via Streams (no polling ever):

And thats just the beginning. There are so many incredible ways to use Macrometa to create fast, powerful, data driven apps and APIs that run worldwide automatically such as graph processing, stream processing and serverless functions (coming soon using JavaScript WASM and V8 isolates). BTW, everything you see above can be done via our 

There's a brave new world of distributed apps and APIs waiting to be built by you for 

. It's time to say Goodbye Cloud, and Hello Edge with Macrometa.

 and get started in minutes. You can get all the 

 and we're a chat box away (to the bottom right of the macrometa web page)

Create Global Apps & APIs In 5 Mins With Macrometa's Seamless Architecture

Today, June 12th, 2019 marks a very special milestone for all of us at Macrometa. Starting today, anyone can 

 for a free developer account and build geo distributed apps and APIs that run across our network of 25 global cloud data centers.

Macrometa for building geo-distributed apps

While the idea behind Macrometa had been knocking around in our head for a little over 5 years, turning all those napkin sketches, sticky notes, midnight conversations and 

 took more than 2 years and the combined efforts of many talented and passionate engineers on our team in Palo Alto California, Paris France, Sofia Bulgaria and Pune India.

One of our most cherished ideas at Macrometa is that anyone should be able to build a globally geo distributed application or API without knowing any more than they already know about using databases with languages like JavaScript and Python. The era of needing large cabals of Phd's with years of expertise in building distributed systems to get correct, accurate distributed systems behavior with high performance needs to end.

We set out with a goal to democratize the platforms that until now only companies like Google, Facebook and Amazon have been able to build. With Macrometa - anyone can build apps and APIs that run distributed, cross region and multi cloud and they don't need to know anything about concurrency, consistency or mastering the arcane complexities of distributed systems.

PS - like our Octopus mascot? Her name is Calico and if you want stickers of her for your laptop, drop us a note at justin@macrometa.com

Build Geo-Distributed Apps - Hello Globally Distributed World!

Contributed by EDGE GRAVITY from their new blog series: 

. This collection of blogs will focus on new technologies and industry leaders in the 

 as way to educate a broad IT audience in the future capabilities at the edge.

Macrometa Global Edge Fabric is an Edge Native Platform as a Service (PaaS) for building distributed applications and APIs for the web, mobile devices, 

, gaming and more. It delivers instant, low 

 stateful data for Single Page Apps (SPAs), Progressive Web Apps (PWAs), and more.

Chetan Venkatesh, CEO at Macrometa on how edge will generate an entire new class of services based on proximity to customers, location awareness, and the ability to co-ordinate data and services that don’t work in a centralized cloud. The edge requires a paradigm shift from the current architecture of databases as edge demands the processing of massive amounts of data in real-time to present the customer with instant responses.

Challenges of edge computing for application developers:

Real-time data analysis at significant volume

To see more thought leadership videos from the Global Edge Forum event visit the EDGE GRAVITY YouTube playlist: 

Reprinted with permission. Original source at 

Building Applications For The Edge With A Datacentric Model

Life happens in the HERE and in the NOW. The HERE and the NOW are defined by where we are and what interactions, events are going on around us and with us. These events and interactions occur in the real world and not some far off data center a thousand miles from us. Similarly every interaction we do typically involves “state” or data about the interaction that combines with other state to create “context” which is essential for us to create and understand meaning from all the interactions we have . But we store this state (data) 1000s of miles away in far remote locations - cloud data centers.

This is like interacting with someone who either doesn’t remember from one sentence to the next sentence (aka stateless) or someone who has to travel 1000s of miles to respond to us each time (aka centralized state stores). If we put it this way, it certainly sounds silly, doesn’t it?

But this is what is presented as a normal way of doing things at the edge with stateless apps and centralized databases. There has to be a better way!

 and streams that provide data at the edge, can scale across tens and hundreds of locations worldwide, and yet act in concert to provide a single coherent multi master database or stream.

It means we need to design systems that work on the internet with its unpredictable network topology, maintain causality over networks that are lossy, and avoid centralized forms of consensus - while still arriving at some shared version of truth in real time. Can this be done? This is the the question that piqued our interest and started a new journey at Macrometa.

There are many good distributed database (sql, nosql,..) and streaming systems out there. So why can’t these systems enable us to have state locally at the edge?

Note: There are some NoSQL/NewSQL databases that can span 3 or 4 or 5 regions, but are not exactly a true geo-distributed or edge native database.

 is something that allows reads and writes locally in all locations in parallel and does not require a user to magically know which data should be placed in which location or require that user to redesign the schema every time they want to add/remove a location.

Coming back, so why can’t the current generation of databases and streaming systems handle multiple locations?

One problem with most of the current distributed databases and streaming systems is that they rely on consensus protocols like Zab (zookeeper), or Raft or Paxos as their foundation. These are certainly beautiful protocols; but these protocols are meant for running inside data centers where the networks are ultra-reliable and low latency.

These protocols (and thereby the systems built on these) are not a good fit for WAN environments where the network is lossy, jittery and high latency. What we need database and streaming systems that are coordination-free and offline-first.

Let’s say you go to a nice restaurant with your partner and order a wine and dessert. Let’s say you ordered wine from the bartender and your partner ordered dessert from the waiter. How would you feel if the restaurant serves you nice folks only one i.e., wine or dessert but not both?

It certainly does not make sense. But that is how most current generation NoSQL databases work with their last-writer-wins approach. In other words, when two edge locations receive writes in parallel (i.e., not causally related), then only one change gets picked up and other change is lost.

What makes more sense is for both of your orders converged and the restaurant serves everyone both wine and dessert. In other words, what we need are databases that can intelligently converge changes in real-time at each location without coordination, but all come to the same definite version of truth.

If interested you can look at CRDT research for more info on this. CRDTs are objects that can be updated without expensive synchronization/consensus, and they are guaranteed to converge eventually if all concurrent updates are commutative and if all updates are executed by each replica eventually.

Similarly, there are some use cases like the double-spend problem where strong consistency is a must. But that does not mean that other use cases are penalized to support this subset of use cases. What we need are databases that can support various levels of data consistencies so that developers can make their choice depending on the use case.

Imagine you went to a coffee shop with your colleague. Your colleague orders one `Cafe Latte` Tall. You order something like below :

1/4 teaspoon vanilla extract - mix in after cycle is complete

You can imagine how the coffee barista looks at you. You might get a similar look from your colleague as well. The reality is both you and your colleague ordered the same thing i.e., `Cafe Latte` but what is different is the abstraction level of the order.

In recent years, there is significant progress on CRDTs and CRDT related technologies. But much of the research is about data types like Counters, OR-Sets etc. That probably makes sense from an academic perspective. But what developers need is `Cafe Latte`. Not the ingredients.

What is needed are higher level abstractions that developers can work with and quickly be productive - like databases with various consistency semantics, query languages etc.

When you choose a database today, you’re not choosing one piece of technology. You’re choosing three i.e., storage technology, data model, and API/query language.

For example, let’s say you choose Postgres. What you are choosing actually is the Postgres storage engine, a relational data model, and the SQL query language. Similarly if you choose MongoDB you are choosing the MongoDB distributed storage engine, a document data model, and the MongoDB API.

A fundamental problem with many of the current generation databases including the above mentioned ones is that they are below big monolithic systems with features interwoven between all of the layers. Monolithic databases continue to be useful pieces of technology, but they are evolutionary dead ends. They are the products of a smaller, less-connected era, and their shortcomings risk becoming liabilities for your business as it evolves.

Document databases, column-oriented, row-oriented, JSON, key-value, etc. all make sense in the right context, and often different parts of an application call for different choices. This creates a tough decision: Use a whole new database (and licenses, etc.) to support a new data model, or try to shoehorn data into your existing database.

What we need is layered model that decouples its various layers - data storage, data organization, data manipulation and data model.

This allows a single database system to support multiple data models against a single, integrated backend. Document, graph, relational, and key-value are examples of data models that may be supported by a multi-model database.

Support multiple data models against a single, integrated backend,

Employ a unified query language to retrieve correlated multi-model data such as Graph-Json-Key/Value etc in a single platform and

Support multi-model ACID 2.0 transactions.

Data privacy is a big issue and rightfully so. GDPR is one big step in that direction and I would imagine more regions and countries will follow this path in future. Database and Streaming systems (being holders of data) fall right into middle of this.

Most databases and streaming systems currently do not address this. The databases that attempt to address this typically use techniques like Geo-Partitioning (break the dataset into multiple subsets or partitions) and keep each subset in a different region.

While on the surface it looks right, this approach creates few problems. For example, it requires customers to know which subset of the dataset will be used in which region and then partition the dataset so that the partition falls into that region.

Similarly if a business is doing well and their customer base expands to other regions, this approach would require deep schema changes to support proper placement of data.

What is better? A database or streaming system that can geofence the data natively with strong isolation and does not require schema changes when a business expands to more regions.

The above are some of the fundamental issues with current database and streaming systems. We cannot address these by doing more of the same. We need to build systems with architectures that address these problems natively. Is it possible?

Certainly yes! We have built Global Date Network that addresses the above and other issues natively. It is currently running in 25+ locations worldwide with scheduled expansion beyond 100 by end of the year.

Clearly, more databases and streaming systems will emerge to address these fundamental problems. The important point is that the technology is available today, and provided by 

Why Global Data Network? Or Why We Built A Truly Coordination-Free Distributed Database For Edge Computing

“Do we need a next generation of game consoles?”

Guest post by Gaming Industry veteran Justin Keeling who has held senior leadership roles at Fox Interactive, Disney, and many other amazing industry leaders.

GameSpot recently polled their readers and asked them the following question:

With 5G on horizon and end-to-end fiber optic pipelines, how close are we to a future where gamers can just stream everything?

75% of readers chose: “Not remotely close”.

The skepticism is understandable. Since 2009, cloud gaming platforms like OnLive and Gaikai have been hyped as the disrupters of the traditional gaming console. I played the first OnLive demo at GDC 2009: Crysis running on a mid-range laptop. Most gaming desktops couldn’t run the game at the time and OnLive felt like black magic.

Outside of the demo environment - and in the real world- it couldn’t live up to the hype. Neither OnLive or Gaikai still operate, although some of their core networking technologies made it into current platforms like PlayStation Now.

But even a decade later nothing yet offers the invisible controller-to-cortex response that most games require.

Over the next 18 months, the game streaming experience is set for a resurgence. Microsoft’s xCloud and Google’s Project Yeti have fired the first shots by leveraging their parent companies’ end-to-end ecosystems to build platforms designed around streaming. Amazon and PlayStation are also cooking their own solutions. And all of the top five game publishers are betting heavily on a game streaming future.

What’s different today, is that an industry-wide critical mass of telcos, CDNs, hardware makers, tool providers, content creators – and crucially– a new generation of deep tech researchers are working in coalition to finally make game streaming a reality.

Edge computing is one of the jigsaw pieces needed to make this all work. Simply put; the Edge is a post-cloud network architecture that is hyper-local, decentralized, and much more efficient than the current, monolithic cloud infrastructure that is behind the current digital ecosystem. And the implications for gaming go far beyond just streaming latency.

I’ve been advising Macrometa for a little over a year to understand the implications of their offering to the 

, and I’m excited that Macrometa is attacking one of the most significant parts of the problem at the intersection of 

. This holds the promise of radically transforming the experience of both gamers and game developers. Macrometa has recently emerged from stealth with its planet scaling serverless edge cloud – the Global Edge Fabric. I had an opportunity to speak with Joshua Petty, Macrometa’s Vice President of Products about their potential impact on gaming. He quickly agreed that edge computing offers incredible potential to easily create this new breed of streamed games.

“The team realized early on that there were many benefits to relocating game processing, logic, and analytics closer to the end user. Burying the infrastructure in traditional cloud architectures and databases is costly and severely limits the ability to create truly global, immersive environments for all users. What was missing was a powerful data and messaging platform at the edge that allowed developers to quickly create applications and games using a pre-built and flexible backend for managing dynamic data and content. So we decided to build it as a simple service for developers to consume. Moving everything closer to the end user was the obvious approach to resolve issues of performance and scale.”

“We also realized that improving the experience of developers required more than just providing a global multi-model database or data platform. We went to work at making the developer experience simple and seamless, combining a flexible and globally scaled edge database with serverless functions, containers, Streams that unify pub-sub messaging, and the ability to expose your data models and content as GraphQL or RESTful APIs automatically. The result is a simple backend delivered as a service supporting Unity and Unreal engines, along with java, Python, and other common web languages and runtimes.”

This type of approach could have a significant impact in the near term on some of the bigger challenges faced when attempting to build a ‘streaming everything’ game experience.

Millions of gamers play Fortnite, but the experience is segmented across different platforms and multiple devices. Recently, progress has been made by stitching together some elements of these ecosystems (thanks Microsoft, Nintendo, Epic).

However, Fortnite is a special case. It’s the world’s biggest game, the users have spoken, and “stitched” is the operative word here. PlayStation is still holding out. For most other games, forget it.

Users clearly want the Netflix & Spotify account experience. Cueing a track at home on an Echo, transitioning seamlessly to phone during a commute, and back onto a laptop at work.

The decentralized architecture at the core of Edge Computing, and platform providers like Macrometa that eliminate backend complexity for developers will help to enable this same seamless user experience for high bandwidth games.

2. Next-Generation Match-Making & User Management

There are plenty of back-end-services out there for multiplayer gaming. If you’re looking for the basics of match-making, leaderboards, and friends lists you’re already covered.

Larger game communities are magnitudes more demanding and require custom built tools – entire studios dedicated to live ops management and custom network architecture.

For example, Activision subsidiary Demonware helps build the multiplayer architecture for millions of concurrent players throwing mountains of real-time data at games like Call of Duty and Destiny 2. Under the current limitations of a centralized cloud/CDN internet architecture, it is a miracle these teams have made it work as well as it (usually) does.

But often it doesn’t work well, especially at scale. Destiny 2’s matching has been the source of community frustration for years. The game’s sluggish inventory systems are constantly lagging. Meanwhile, the PlayStation Network achievement ecosystem is bewilderingly slow to sync. Legacy cloud-based database architectures simply can’t keep up with increasing data and state management demands for much longer.

We need a better core solution for this. As we transition into the upcoming generation with more ambitious game designs, more users on the network, and more data flowing through that network, systems built on hyper-local Edge architecture can dramatically improve match-making accuracy and multiplayer user experiences across the board.

As much as we love Pokémon Go, the sheer volume of users and location data made for a laggy launch experience. It still gets sluggish around high data focal points.

Edge Computing’s architecture is tailor made for a new generation of challenging, high bandwidth AR games. Distributed platforms like Macrometa let developers synchronize player activity in much smaller groups based on location data, and then can automatically trigger functions that respond to local gameplay actions on a peer-to-peer basis.

The states of all in-game entities are also stored in real time at Edge locations much closer to the user than current centralized cloud servers, letting developers execute much more complex AR simulations, and with much less lag.

4. New frontiers in Real-time Data driven gameplay

 can sync local game world and player states in real-time, this also has huge implications for game design. If you’re designing experiences with a fixed, or intermittently updated version of what the game world is and what the player’s doing in it, you’re limited to a much less dynamic set of options

Edge Computing also manages data not just in real-time but in context– designers can use this contextual data to create custom experiences based on actual player progression. That opens up exciting new possibilities for truly persistent game play in online worlds, while actually lowering the processing and network overhead on a gamer’s console or device back home.

Archive