Explore insights on real-time edge computing, AI, and cloud solutions. Learn how our technology optimizes performance across industries.

Blog

We’re excited to announce the general availability of 

Macrometa’s complex event processing (CEP)

Complex event processing is a tricky topic, made simple by 

 giving an overview of CEP and the distinction between CEP and streaming analytics. 

CEP allows you to work with one or more data streams, optionally combined with data-at-rest, to infer “complex events”. For example, you might have streaming financial trade data, and by joining that with other data about macroeconomic conditions, you could infer a “market correction” event.

Stream Workers provides typical streaming analytics capabilities, including joining streams, aggregations, counts, and filtering as well as sophisticated high-level primitives for 

 such as pattern recognition, conditional logic, and flow control. The Stream Workers engine runs within the Global Data Mesh, allowing you to incorporate operational data-at-rest with streams of data from both Macrometa’s streaming collections and other event streams, such as from 

 topics. Stream Workers can write to data stores within the Global Data Mesh or stream results to other streaming frameworks like 

Stream Worker's applications can be queried directly using a REST API

. Since Stream Workers is tightly integrated with the 

, there is no need to transfer data from your operational data store to the CEP engine. And they are, ultimately, making large amounts of reference data available instantly for sophisticated predictive modeling at very 

This may sound very complicated and, until now, it has been, but with Macrometa Stream Workers, all developers need is basic SQL and Javascript skills, and the Stream Workers engine handles the rest. In addition, your application can run simultaneously at over 175 locations worldwide and synchronize automatically at the speed of light. As a result, companies and developers using the Macrometa Global Data Network have a real competitive advantage with access to a powerful distributed 

 engine without the developer and administration complexity you’d expect.

Stream Workers can write self-contained microservices and traditional streaming analytics applications. Using Stream Workers, you can create 

, create microservices, and serve the logic of an application backend. To demonstrate this, we’ll show our cryptocurrency trading demo below. Stream Workers can solve so many problems, but here are some simple use cases:

Transform data from one format to another, for example, from XML to JSON, and clean and filter your data:


) data from log files and scrape HTTP endpoints.

Obfuscate sensitive information by filtering it from messages, logs, or other data streams that may include Personally Identifiable Information (PII) or Personal Health Information (PHI).

Correlate data by joining multiple streams to create an aggregate stream.

Derive insights by identifying event patterns in data streams:


Summarize data with time windows and incremental aggregations

Trigger actions based on streaming data, either as a single service request or a complex enterprise integration flow.

Build and refine predictive models for machine learning

Make real-time offers to customers who are visiting an eCommerce site based on browsing or shopping history

Threat detection in cybersecurity applications

Stream Workers Demo: High-Frequency Crypto Trading App

 to trade Bitcoin locally in US, Germany, and Japan exchanges.

An exchange arbitrage is a trading strategy based on the differences between the price of the cryptocurrency at different exchanges. Generally, these opportunities open up if there is a price discrepancy, and the contrast accumulates over time to finally become a significant amount, generally 1–3%. However, these arbitrage opportunities typically only exist for seconds, so an arbitrage trader must constantly search for the best opportunities and implement them immediately.

Implementing arbitrage strategies in the crypto market takes work and involves several technical complexities. Besides having deposits in multiple major exchanges, on the technical side, it consists of the ability to monitor arbitrage opportunities simultaneously, making trades locally at each exchange with minimum delay and timely visibility and interactions between the trading agents distributed globally. The challenge requires big dollars and, until now, a team with the technical expertise to build a distributed infrastructure.

To build the demo, we used the following:

Macrometa Global Data Mesh collections, both streaming and data-at-rest, and 

We use a document store to store trade data and share all trade-related metadata and state between the decentralized, stateless bots and components.

We use geo-distributed streams for publishing real-time price data from each exchange. Each bot subscribes to the stream and decides on whether to buy or sell.

Lastly, use real-time updates from the datastore to notify the clients/bots/subscribers of changes to the database so they can modify their trading strategy.

The entire backend of this application is written in SQL and Javascript with only 174 lines of code! Watch the video for a complete app walkthrough, or check the code on 

Stream Workers logic is written in standard Postgres SQL and Javascript. Our web console has a full-featured REST API, code editor, and publisher. You can query any data store directly in your code without any configuration.

We provide native sources and sinks to connect to Kafka, Google Pub/Sub, MQTT, and S3.

After you’ve written your code and are ready to save the Stream Worker, click save, and pick the regions to deploy your Stream Worker.

When you’re ready to go live, click Publish. You don’t need to manage any software images or configuration; we get your app up and running in a few seconds. Macrometa’s intelligent traffic routing means every time a user or client sends data to the GDN, and it’s routed to the region closest to them. That means your applications will be FAST.

We provide several sample Stream Workers samples in the web console to get started quickly.

We think Stream Workers is ushering in the next era of developing applications at the edge, but don’t just take our word for it. Instead, read through our documentation to see all the available APIs, and 

 to try them out. We can’t wait to see what you build.

Implement Complex Event Processing in Minutes (Macrometa tutorial)

Using Macrometa Stream Workers In A Real-Time Log Analytics Dashboard

Announcing Stream Workers: Complex Event Processing And Streaming Analytics At The Edge

 now includes native support for SQL. This blog post discusses how we did it (and why it is important) and then digs deeper into how to write queries.

 (GDN) is a Platform as a Service for building and running applications at the edge. Run regionally, globally, or both - the GDN automatically synchronizes data between locations in milliseconds. Build new applications or extend existing applications to improve performance and comply with data locality requirements. We provide data and compute infrastructure from more than 175 locations globally, reaching two billion users and ten billion devices in 50ms.

 in our latest release, developers don’t need to learn a new query language to harness the power of the platform.

Historically, SQL has been used to manage relational databases on-premises, and more recently it powers many major legacy cloud databases, such as Snowflake and Amazon RDS. SQL has been with us since the 1970s and is the database query language powering most websites, most business logic, and business intelligence applications in businesses both large and small across the globe. SQL is by far the most popular language in the world for querying databases. When we compare SQL with any general-purpose language by usage, it sits around tenth place. (

 SQL sits in fourth place and is ranked above general-purpose language giants such as C# and Java.

The Macrometa platform is a distributed edge-native platform that runs across multiple data centers. Most real-life applications are stateful and our platform maintains state across up to 175 locations globally, allowing you to automatically synchronize session state and any other data you store in the Global Data Mesh, all at the speed of light.

In the latest release, we have added support for the Postgres SQL query language in the Macrometa Global Data Network. With this addition, SQL - which has been traditionally used with monolithic platforms - can be used with the distributed Global Data Network. There’s no learning curve because developers can continue using the tools and language they are comfortable working with.

The Macrometa Global Data Network provides very low latency and very high performance for your applications. You can get productive on Macrometa’s platform right away by using standard SQL. We also support the Macrometa native query language, DynamoDB APIs, and Redis APIs.

 and be sure to check out the rest of our 

The Macrometa Platform Now Supports The SQL Language

We’re excited to announce the general availability of Macrometa Search in the 

. In addition, Macrometa Search is now available to all Macrometa customers.

Full-text search is a powerful capability that opens up many use cases but is less widely adopted than it might be because traditional search engines such as Elasticsearch and Solr have required data movement, complex search engine management, and developer search expertise. Macrometa Search takes the complexity out of the search. Getting started with Macrometa Search is as easy as writing a SQL query, with no data movement is required because Search Views index your data in place in the Global Data Mesh; no separate search engine to administer because Macrometa Search is a part of the 

, and with many advanced search capabilities for developers who are search experts.

With Macrometa Search, developers can build powerful search-powered apps by creating Views on existing Key-Value, Document, and 

. There is no need to spin up a new search cluster and build a data pipeline – all you have to do is configure a Search View across collections in Macrometa and start running queries.

Search indexes and data are automatically synchronized across any number of chosen locations (over 175 regions globally). This brings the power of search close to your users for lightning-fast query responses. Search Views are automatically synchronized across geographic locations without any configuration or management required, so your user who is in New York today and Singapore tomorrow still receives the same low latency, customer-delighting experience from your application. The GDN’s global 

 is less than 50 ms from over two billion users and ten billion devices worldwide. So, your apps will be FAST.

What can you build with Search in Macrometa?

Search allows you to perform instantaneous and sophisticated full-text search queries for 

 in documents across multiple linked collections. Each search view represents an inverted index that contains the search configuration and all document attributes in the linked collections.

You can facet queries by document attributes, sort the results by relevance using popular scoring algorithms such as TF-IDF and BM25. Macrometa Search supports fuzzy search, correcting typos and spelling errors. Macrometa Search supports analyzers for text in multiple languages. Even if you don’t need the full power of Macrometa Search, you can call APIs to leverage Macrometa Search for stemming, tokenization, n-grams, and other search indexing functions “a la carte”. Macrometa Search provides powerful advanced features like geospatial search and graph search in addition to standard search engine functions such as pattern matching, range queries, token search, phrase and proximity search, and faceted search.

It’s now straightforward to add search functionality to a website. Still, you can also do things like add location awareness to an app (to find a restaurant close to you, for example), have an up-to-date inventory for your eCommerce store search, or power a real-time analytics dashboard with up-to-the-second data - the possibilities are endless.

So, how easy is it to add a search to an app? Let’s check out a demo to find out!

Search Demo: Extending the Address Book Quickstart app

In this demo, we will add search functionality to our 

, an address book. The guide walks you through creating a complete globally distributed backend with a CRUD API and native data streaming in less than 10 minutes.

We have already created a datastore, added some data, and created 

 We have a pre-built front end for you to use.

We will create a Search View, define which fields we want to index, and create an API with a Query Worker to power the search field on the app's front end. Check out this video.

The Macrometa Search developer experience

Building search functionality with Macrometa is surprisingly easy compared to traditional methods. You're ready to go if you have data already stored in Macrometa. You can also create a cache of an existing database to bring your centralized data to the edge and add search, vastly decreasing user latency and improving the user experience.

Once you have data to search, create a Search View and define what to index among your collections.

The index will populate nearly instantly depending on the size of the dataset. Once the Search View is ready, you can interact with it by 

 through any of Macrometa’s supported query languages as if it were a collection or through the 

. You will have immediate access to a powerful search engine with advanced features such as geospatial searches and graph searches right out of the box.

One of Macrometa’s guiding principles is to make every developer a hero by making simple what was complex. Full-text search is a powerful feature that opens up many use cases but is not as widely adopted as it might be because traditional search engines have required data movement, complex search engine management, and developer search expertise to get started.

Macrometa Search is a powerful product for search experts, and all of the tuning and power user functionality is available. Best part? You don’t need to know about any of that to get started and start realizing value from Macrometa Search immediately. We can’t wait to see what you build.

Beyond Text Search: Exploring Advanced Search Capabilities With Macrometa

Expand Your Search Capabilities With The Macrometa Global Data Network

Announcing Macrometa Search: Powerful Full-Text Search, As Easy As Writing A SQL Query

We’re excited to share some big changes happening here at Macrometa. Over the past 6 months, the Macrometa design team has been hard at work crafting a new visual aesthetic, brand, and design system that touches every aspect of the company, as well as completely rebuilt web application. This has been a labor of love, and is still very much ongoing. But today, I would like to share a look at what’s new, along with a peek behind the curtain at some of our tools and methods.

First and foremost, I’m delighted to introduce our new logo, one of a kind that defines us.

The octopus has long been a cherished creature by all of us at Macrometa. I mean, who doesn’t love the noble octopus? They’re such marvelous beings; incredibly intelligent, adaptive, and elegant.. an evolutionary masterpiece by all accounts. We also love the octopus because it’s a perfect metaphor for the Global Data Network.

For example, the brain of an octopus is fully distributed across their entire body, with each limb containing its own independent “mini-brain”, much like how the GDN is a fully-distributed active-active architecture with no single point of failure.

But I digress, we’re here to talk about logos and pixels! Read more about our love for the octopus here: 

With our new logomark, the beloved octopus finally has a home front and center in everything we do. This mark brings clarity, symmetry, and confidence into balance with the playful, fluid nature of the octopus. It’s unique and yet quite familiar, offering a bit of personality and energy wherever it goes.

The octopus can be a rather challenging creature to incorporate into a logo, simply because they’re so amorphous and pliable. It becomes an exercise in balancing several contrasting attributes: how detailed vs. abstract, how cartoonish vs. realistic, and how cute vs. serious? Too much or too little in any direction and the concept misses the mark entirely.

Our new brand typography is centered around the 

, a geometric san-serif family by type designer and typographer 

. I love this typeface. Averta combines the stab reliability of rounded geometric glyphs with an open, refreshing playfulness. It also looks great for large headlines while maintaining sharpness and legibility at smaller sizes.

Early days of the Macrometa Design System (and a new Web App too!)

Over the past several months, our engineering and design teams have been hard at work building a new and greatly improved web console. This app serves as a primary touchpoint for all users of our platform. Our web console is by every measure a mission-critical application and was previously written in 

If you’re familiar with BackboneJS, you will have some sense of what we were up against. If not, BackboneJS was an early JavaScript application framework based on the MVC architectural paradigm. When BackboneJS first launched in 2010, it brought a huge leap forward for authoring interactive web apps. But alas, the time has marched on. We considered a handful of ways to port our app away from BackboneJS incrementally, but in the end realized we needed to bite the bullet and rebuild the entire thing from the ground up. And so, that’s exactly what we did.

, and all the wonderful open-source conveniences one might expect under the hood of a modern frontend stack. This new app architecture has already vastly improved our development velocity, which will soon become evident with more frequent and featureful releases. Having worked on a lot of React apps in my time, I can’t tell you how unnerving it is to suddenly be sent back in time 10 years, losing access to all of the amazing tools and design patterns that have happened since. Let’s just say, it’s good to be back!

Of all the enhancements under the hood, I’m most excited about the addition of 

 as the foundation of our UI component library, and how this has allowed us to move fast while unifying the presentational layer of our built applications with our design tooling and resources. I’ve used Chakra UI on several occasions prior to joining Macrometa, I really can’t sing its praises loudly enough. I highly recommend taking it for a test drive, if you haven’t already.

Like many design teams, we work almost exclusively in Figma, and have produced a comprehensive collection of foundations, components, patterns, and guidelines to support and streamline our work. The mocks we produce in Figma use components, styles, and spacing rules that correspond precisely to our Chakra UI component library and theme configuration, leaving zero room for divergent or ambiguous details. The official 

 was a huge help in bootstrapping our initial component library and allows for easy customization as well.

This setup helps our relatively small design team move much faster and more effectively while helping our frontend engineer teams implement our work faster and more accurately.

We’re still very much in the early days of building the Macrometa design team and are always looking for help. If that sounds like fun to you, feel free to reach out and say hi!

Macrometa’s New UI And Design Process - Unravelling An Aesthetic Pleasure

Hi and welcome to Macrometa’s inaugural Developer Week. I’m Chetan, Macrometa co-founder, and CEO. This week is the beginning of something very special and something a long time in the making and that is the reason that I and my 100 plus colleagues in 14 countries are incredibly excited to share Macrometa with you.

Macrometa is a new kind of platform for developers for building an entirely new generation of advanced distributed microservices using our 

 (or GDN) - a hyper distributed edge cloud available across 175 edge locations around the world. The GDN lets developers like you create real time data powered services that run closer to their users and their devices in edge locations (or PoPs - short for point of presence) around the world, within 50 milliseconds of access to more than 90% of internet connected humans and their devices. 

As a developer platform, it’s important for us to build and create out in the open. Our company is built on the values of Honesty, Humility and Heartfulness. And our mission is to make every developer a superhero. We want to share knowledge, share ideas, but most importantly, we want to be available, transparent and open to the developers that use Macrometa today - and those that may in the future. Having a community and being heard are deeply important and fundamental to our mission, and we’re pursuing both this week with sincerity and honesty.

There are a lot of things to be excited about here at Macrometa, and we want to share all of those things with you. So whether you’re a curious observer, a power user, or just looking to learn, we’re happy you’re here and look forward to seeing and hearing more of you in the future.

Watch the announcement video below or keep scrolling.

My cofounder Durga and I started Macrometa in Nov, 2017 with a handful of talented and wide eyed technical dreamers to realize an idea that we both had quite accidentally been pursuing for the past 20 years - an idea of building a distributed architecture to efficiently solve the challenge of keeping data in lots of places in synch with each other. We both had worked on consistent replication and synchronization techniques for all kinds of interesting problems. And invented lots of unusual ways to solve this problem across the data stack - (I have 19 patents in data related inventions alone and Durga has invented way more than I have – but he brings the humble to our values and doesn't talk about them much).

Our young and small team envisioned a world where developers could build distributed cloud apps that were free from geographical constraints and free to run concurrently in lots of places in parallel - the challenge being keeping data (or stateful data) in synch between all these locations so that compute logic would always see consistent state. This problem has vexed computer scientists for decades who have largely accepted that the C-A-P theorem forces a tradeoff between having good consistency and sacrificing throughput (and performance) due to coordination protocols needed to guarantee consistency, or no consistency (LOL eventual consistency) and simplistic conflict resolution like last writer wins to achieve a level of throughput and practical performance required to run modern apps.

To cut a long story short - Durga and the team invented a new coordination free protocol that uses Conflict Free Replicated Data Type (CRDT) we call Coordination Free Merge Data Transforms as its underlying substrate (read more about it on 

). The next set of challenges we faced were in making this radical new protocol and architecture work at Edge Scale.

Where a typical cloud is really a handful of data centers - maybe 15 -20 regions in total and a customer using maybe a couple of regions (mostly for backup or high availability) - the edge is defined by extreme scales of needing apps and data to be available, replicated and synchronized and served from potentially 100s of regions for most use cases and even thousands of regions for some use cases. To do this, the team designed and built a global network (with a Content Delivery Network aka CDN like topology). We called it the Global Data Network or GDN, with a 

 to connect, ingest, store, serve, and process data. With this architecture, developers can turn data into APIs, 

 functions and Containers to perform stateful computing at hyper proximity to data producers and consumers. We also set out to solve emerging challenges with data regulation and legal frameworks for data sovereignty with our built-in, in-region 

. The end result: a developer platform that is uniquely built to solve the problems of real-time data and operational analytics in a way that the cloud just can't.

And in reading all of this I hope it's obvious that our team is deeply passionate about developers, and have a clear vision for where technology is headed, and how the edge in particular can become a place as important as the cloud for where you build and run applications.

 is, “The future is edge native: Building on the cutting edge with Macrometa and friends.” So what do we mean by that? And why should you care? Allow me to show you.

The edge has been for the past few years - the shiny new thing, where it’s only been possible to build and run what we like to call “toy and trinket” apps leaving developers frustrated because it lacks the sophistication of the cloud primarily because the ability to store and manipulate stateful data at the edge has been missing. Most edge platforms in the market have hacks for stateful data - providing multi-region reads but single-region writes, or poor consistency protocols that make them impractical for real world use (such as not being able to update or modify a key-value pair more than once in 2 seconds).

We’re not interested in building toys here at Macrometa. We’re not a Fisher-Price platform for kids. Neither is this a Barbie dream house or a Castle Greyskull to idle away an afternoon or weekend building a toy or trinket app on. We’re giving developers the seriously powerful, enterprise grade, mission critical platforms and developer tools needed to build real world apps and APIs that developers can rely on to work well and work at scale. All with extreme levels of performance.

We’re teaming up with a few incredible partners such as Akamai and DISH Wireless, as well as amazing customers such as DevCycle, Resulticks, AppSecEngineer and Arrive.gg this week (that’s the “and friends” part) and will tell their stories about why and what they have built on the edge with Macrometa, and what you can do with it, and what’s in the future. We have demos, thought pieces, videos, giveaways, and other exciting things planned all week long and we hope that you will come and join our celebration of learning, growing, ideating, and building impossible apps on the edge with Macrometa and Friends.

I know I already said we’re not for kids, but we’re also celebrating Macrometa’s 5th birthday this week 🎉

Macrometa has grown into more than a hundred people around the world in 14 countries since those early days of November 2017. And on behalf of all of them, and my amazing co-founder and CTO Durga Gokina, I want to welcome you again to 

. Thank you for your time and support - see you on the edge.

Welcome to Developer Week Day 1

First principles thinking is a powerful but under-used approach to building systems to solve complex problems. The basic idea of first principles thinking is to break down complex problems into essential elements that you know are true and then reassemble them from the ground up to solve the problem. In my experience, this approach enables one to unlock creativity and build innovative and robust systems.

Why do folks ignore this? Or maybe folks don’t know how to break problems into first principles and reassemble novel solutions to the problem. I had excellent results with applying first principles thinking to solve complex problems.

So thought I would share my approach in this post to show how one can build systems (and how we built Macrometa) by applying first principles in two different and complex domains — financial markets and 

Building a Trading System from First Principles

Let’s take investing/trading as it is a hot topic this year with inflation and recession worries. The problem is simple - I want to build a robust and profitable trading system that generates profits with less risk & not require too much of my time. Specifically, financial markets (equity, bonds, currencies, commodities, etc.) are complex and vast, so our task is to navigate this vast complexity and solve our problem.

Most retail investors have strategies to trade/invest in the markets. If we distill those strategies, often the first principles unwittingly are:

I’m sure we can do a little better and decompose financial market behavior into three first principles.

First principles for building a trading system

In any time frame (daily, weekly, monthly, etc.),

Each market can move only in three ways: up, down, or sideways

Some markets move opposite to other markets

Some markets move faster than other markets in the same direction.

The above principles are simple and obvious, isn’t it? So, how do we build a profitable trading system from these first principles?

Each financial market can move only in three ways - Up, Down, and Sideways

Let’s take the first principle - Each market can move only in 3 ways: up, down, or sideways. Assuming we trade only on the long side (i.e., no shorting), it means we want to be in the market when it is going up and get out of the market when it is going down.

One way to do this is by using moving / rolling averages. Mathematically, a rolling average of the price will always be below the price, when the price is going up and vice versa when the price is going down.

Buy Rule - Buy equities when the market closes above its moving average.

Sell Rule - Sell equities and go to cash when the market closes below its moving average.

Some markets move opposite to other markets.

Let’s take the 2nd first principle - Some markets move opposite other markets. We know markets go through risk-on and risk-off periods. During risk-off periods, the capital flies from risk investments like equities to safe assets like US Treasuries, Bonds, etc. You can measure this behavior via rolling correlations.

Sell Rule - Sell equities and go to bonds when the market closes below its moving average.

Following are the results of a trading system using the above rules.

Equities Proxy: VFINX (Vanguard S&P 500 Index Fund)

Bonds Proxy: VBMFX (Vanguard Total Bond Index Fund)

Moving Average: 10-month Rolling Average.

Test Duration: 1990 - 2022 (September). Timeframe: Monthly.

One can undoubtedly develop additional first principles to improve this system's results further. For example, using the 3rd principle to compare bonds and cash further reduces risk. You can also try this with other moving average lengths to see the stability of the returns. Finally, some skeptics might question why we’ve used a monthly timeframe and indexes are used in this system. The reason is simple - Inversion Mental Model.

Disclaimer: Anything in this post should NOT be taken as financial advice. Use it at your own risk.

Building Macrometa GDN from First Principles

 is a platform for enterprises to build global systems of interactions and APIs to serve users around the globe instantly. In the past, only a handful of companies, like Google, Facebook, etc., could build these types of systems utilizing an army of PhDs and millions of dollars. Macrometa GDN gives enterprises an easy button to harness their data in real-time and, eventually, contextualize, replicate, monetize and share at a fraction of the cost and expertise.

A few years back, when my co-founder (Chetan) and I set out to build GDN, we faced a challenge: creating an edge native data & compute platform running across the globe in 100s of locations. Further, this platform must run seamlessly across multiple clouds, on-premise data centers, and telecom (5G) locations. None of the existing data or compute platforms could do this.

So our first step was to decompose the challenge into fundamental problems (aka first principles) that we know are true and need to be addressed. Then we solved these real problems and combined the techniques to build the GDN and edge compute you see today.

The first principles that underlie the GDN platform are:

Networks are unreliable across data centers.

Networks have large latencies across data centers.

Concurrent reads-n-writes across data centers.

Query languages are CRUD operations on document fields.

Networks are unreliable across data centers

Unlike networks within a data center, the networks across regions are unreliable. So the first step is to address this and create a reliable data replication mechanism across areas. To solve this, we built a replication system using append-only logs with a pull model in the same fashion as 

A key difference - In GDN, the geo-replicated streams are created and managed in a coordination-free manner. Also, initially, GDN used a SWIM-based gossip protocol over UDP to discover regions, but later on, a hierarchical peer-based discovery protocol replaced it. The primary reason is eliminating UDP to reduce headaches and the platform's footprint.

Have you ever tried to access servers running in the USA from Europe or Asia? If so, I don’t need to explain this principle. The results of a simple HTTP ping test between my server (San Jose, California, USA) and various AWS regions are shown below. The network latencies across regions are in triple digits compared to the single-digit latencies we see within a data center.

The challenge is that the current data platforms are designed to run inside a data center with single-digit millisecond network latencies. These data platforms use synchronized consensus protocols like Zabber, Paxos and Raft, etc., which are chatty and require coordination between the regions. So when you run these platforms across multiple data centers, you will have the following problems -

The operations incur significant latencies to complete because of synchronization between regions.

The operations frequently fail because of unreliable networks between regions. The more regions, the more frequent the failures are.

The operations create significant network traffic because of protocol chattiness.

Macrometa GDN eliminates these problems using a coordination-free architecture, i.e., 

 are used for convergence across data centers. More details on this are in the next section.

This coordination-free architecture allows GDN to scale to many data centers. Also, it enables GDN to serve reads-n-writes with low latency and no coordination with other data centers.

Concurrent reads-n-writes across data centers

Conflict-free Replicated Data Types (CRDTs) are data structures that can be updated without expensive synchronization/consensus. They allow data to be concurrently updated on several replicas, even while offline, and provide a robust way to merge those updates into a consistent state. In this regard, CRDTs differ from the algorithms used by Google Docs, Trello, and Figma, which require all communication between users to flow via a server.

GDN incorporates CRDTs by decomposing a database into its most minor elements (aka 

A JSON document is a collection of data types - string, boolean, number, list, or object.

GDN uses RICE (Real-time Intelligent Convergence Engine) to handle insert, update or delete operations on a JSON document. The RICE engine generates CRDT operations for any changes to a JSON document. Similarly, it applies CRDT operations to the local document state and converges to a consistent state mathematically deterministically. The CRDT operations are replicated across all data centers via persistent streams. Similarly, GDN uses Vector Clocks to tag the CRDT operations for establishing causal relationships between changes across data centers.

A big challenge when using CRDT is garbage collection of the ops history log. The history log of a document grows with each insert, update, or deletion of a field in the document. Some platforms try to do garbage collection by coordinating across data centers. But that is not scalable and introduces the problems described in the previous section. Instead, GDN uses an innovative co-ordination free garbage collection algorithm to keep the document (aka history log) from growing unbounded. You can read more about it under the technology section on the 

Currently, only some systems provide CRDT data structures. Providing a whole database built on the CRDT mechanism is more complex. Unlike individual data structures, to give a CRDT database, the database query language (ex: SQL, C8QL, etc.) needs to be translated in real-time to applicable CRDT operations. This is what GDN does to provide a database abstraction to users while using CRDTs and ops logs internally.

For example, an update SQL query like the one below will be translated in real-time by GDN into CRDT operations shown below:

When a user issues an INSERT, UPDATE or DELETE query (aka WRITE queries), GDN converts these queries internally into a series of CRDT Operations. Then these CRDT operations are geo-replicated asynchronously to applicable data centers for local processing. Each data center applies the CRDT transformations and incrementally updates local materialized views. READ queries are served by the materialized view engine for 

. Various types of indexes (hash, skiplist, TTL, geo-spatial, full text ) are supported on these materialized views to reduce the read latencies further.

Similarly, GDN supports multiple data models (key-value, docs, graphs, search) again by leveraging 

 —> A set of JSON documents with a primary key for each document. Supported operations - get, set, delete

Document Store —> A set of JSON documents with a primary key for each document. Supported operations - Query Language (SQL), Indexing & Transactions.

 —> A set of JSON documents to represent a graph (i.e., each vertex and node of a graph is a document). Node documents contain from and to fields. Supported operations - Query Language, Indexing, Transactions, Graph Functions, etc.

Search Store —> A reverse index on a set of JSON documents. Supported operations - Query Language, Indexing, Search Functions, etc.

Most data platforms are monolithic systems. GDN is built as layers, each building on top of the previous layers. The layer model enables GDN to support multiple query interfaces easily. GDN supports C8QL, SQL, Dynamo, and Redis query languages as plugins. More on the way. So the revised picture looks like the one below.

Today most folks use separate data platforms for data-at-rest and data-in-motion. Example - Databases for data-in-motion and 

 for data-in-motion. The data is the same whether it is data-at-rest or data-in-motion.

So why do developers have to use four different platforms (i.e., DB, Streams, Search, Stream Processing) and waste their time wiring these platforms together, spend days to program and ensure the data is consistent across all platforms, etc.? Similarly, why are architects forced into building complex architectures, and why do operators have to manage & maintain a forest of products?

Like most readers of this post, I was that developer & architect. I paid the tax to deliver my stuff. So naturally, this question was on top of my mind while architecting GDN. Consider a table with one document and a stream with one document. Is there any difference between them? No. Say we have an update to the document. The table contains one document with the latest change, whereas the stream contains two documents. In other words, the table is nothing but compaction of the stream contents to reflect the latest state of a document.

In GDN, every table is also a stream. This opens up influential developers' capabilities and eliminates the need for complex architectures that look like spaghetti balls. A developer can now query the data when it is at rest (aka tables) and the data when it is in motion (aka stream processing). Also, a developer can join tables, join streams or join streams & tables as part of their queries. This is a very nice power to have. Please refer to the 

Archive

Announcing Stream Workers: Complex Event Processing And Streaming Analytics At The Edge

The Macrometa Platform Now Supports The SQL Language

Announcing Macrometa Search: Powerful Full-Text Search, As Easy As Writing A SQL Query

Macrometa’s New UI And Design Process - Unravelling An Aesthetic Pleasure

Welcome to Developer Week Day 1

Building Systems & Macrometa GDN From First Principles

Calling All Developers To The Edge - Announcing Developer Week

Say Yes To The Data Mesh: Explore Multi-Model Data Collections And Stores At The Edge

Beyond Cloud Computing: The Advent Of GDN