Explore insights on real-time edge computing, AI, and cloud solutions. Learn how our technology optimizes performance across industries.

Blog

We're excited to share with you that we have launched a public Slack for the Macrometa Community. You can sign up for the Slack 

The Macrometa Community Slack is not replacing our existing world-class technical support. Any urgent issues should still be directed to our support team, and we can help you out.

The Community Slack is for Macrometa users to engage with each other and with some of us at Macrometa. I’m hoping it will come to be a place where our community helps each other solve problems and share best practices.

Of course, any actual problems with the product are for Macrometa to fix, but sometimes you just “get stuck” trying to implement something and benefit from a fresh perspective. In the Macrometa community Slack, we hope you will find that fresh perspective. It’s also a place for the Macrometa community to just “hang out” and get to know each other. I hope to see you there!

See you all soon in the Macrometa Community Slack :)

Macrometa Launches A Public Slack For The Developer Community

Excerpts from this article were originally published in the Datanami article “

Has Macrometa Cracked the Code for Global, Real-Time Data

There is an old saying that time is money. Data has a 

time, location, concurrency, and actuation value

. However, trying to combine real-time and historical data often ends in complex or expensive solutions that may not be able to keep up with the required low 

 demands. In this blog, we will explore the recent 

 about how Macrometa’s architecture and features deliver real-time data value. The article highlights how Macrometa offers a Global Data Mesh, provides a unique use of conflict-free replicated data types (CRDT)s, and integrates the Global Data Mesh with Edge Compute and Data Protection in a “three-layer cake” within the Global Data Network (GDN).

Macrometa CEO Chetan Venkatesh kicks off talking to Datanami about a typical customer example, Cox Communications, and “how the amount of data they were sending in, the scale at which they had to build and integrate these systems made it too complex and slow” with a cloud provider. Once Cox Communications engaged Macrometa, they were able to “reduce costs by 90% and have a picture of reality within the hundreds of milliseconds of latency” vs minutes to hours delay. Let’s run through the Macrometa components that provide these real-time benefits.

A solution for big and fast data - Global Data Mesh

The issue today is that data has to make the roundtrip to a centralized data center - whether on premises or in the cloud - and this is far away from customers, systems, staff, partners, and suppliers. This delay can lower the value of data, if a customer goes to another website with a better response time, or if a logistics company doesn’t see a traffic accident notification. Chetan explains to Datanami how Macrometa addresses this problem with a 

, that is a distributed resource, not a centralized resource like typical cloud providers.

In Macrometa's data mesh model, data can be stored, accessed, and processed where it originates and/or where it is needed. The result is a flexible, ultra-low-latency data layer purpose-built for global, real-time, and event-driven use cases. Data is automatically replicated across the Macrometa GDN, and API requests are routed to the nearest available location. There are over 175 

 around the world. With Macrometa, developers have the tools to build exciting new applications and they can use the data model that makes the most sense for their application. 

“The storage engine is a very small but integral part of our stack,” Chetan said to Datanami. “The real value is in the approach we have for the way we ingest data into the log, the way we collapse that log into objects in real time, and most importantly, the way we can replicate that data across hundreds of locations with transactional and consistency guarantees. That’s always been the missing piece.”

In developing Macrometa future data volumes were a consideration - which could easily be ingesting trillions of events per second per the article. At its heart, Macrometa's architecture is a geo distributed event source with a materialized view engine and 

. Macrometa manages changes at the atomic field level and merges changes made across the network to maintain a single version of the truth via CRDTs, showcasing the power and versatility of CRDTs as a regular JSON database and API.

The article depicts how Macrometa transforms all data changes into a CRDT operation, which is then replicated to all the Macrometa locations using a vector clock. “With a vector clock and a causal tree of changes, Macrometa ‘s system essentially serializes and provides serialization ACID-like guarantees,” as Chetan explains in the Datanami article. “This ensures consistency without having to exchange a lot of messages with different apps.” Macrometa can store, process, and serve data within milliseconds to multiple locations around the world. In contrast, the internal messaging required within 

 can significantly lower performance if they expand past four or five locations around the world.

 ”three-layered cake”: a real-time, geo-distributed storage layer, a compute layer to process data to run real-time apps close to your customers, and a data governance layer to help organizations comply with regulatory and legal standards.

Base layer to store and ingest data - Global Data Mesh

As big and fast data ebbs and flows in all directions, developers and enterprises and everyone that relies on data for decisions need a simplified way to quickly address different types of data. The Global Data Mesh has all the benefits of a NoSQL database with KV, doc, and graph stores - plus Macrometa Streams with pub/sub and messaging queues.

Middle layer bringing it together - Edge Compute

“In our platform, streams and tables are the same things,” Chetan explained to Datanami. “It’s just that streams are real time user tables. And so you can interact with your data in both real time, in-motion fashion with streams, via pub/sub, or you can query it using request-response with SQL.”

What separates Macrometa from other data storage solutions is that it brings 

 closer to people and where data is generated, regardless of where that is. 

 lets developers create simple REST APIs on top of the data mesh. 

 enables Complex Event Processing (CEP) functions and Stream Processing workloads - in minutes instead of days or months. With Stream Workers, data could come from different sources (GDN streams, GDN Database, HTTP endpoints, Kafka, and MQTT etc.) and can be published to different sinks or stores. 

 also allows you to transform data from one data type to another, aggregate streams, and clean and filter data. This CEP engine automatically creates objects like streams, collections, and even tables and remote databases. 

Top layer keeping data separate and safe - Data Protection

 supports geo-pinning or geo-fencing to address data sovereignty use cases. Developers and operations can easily set up 

 for regional and global data, and adjust locations with just a toggle. Macrometa is 

 and offers user, token-based, and API keys 

The icing on the cake - a complete Edge-as-a-Service platform

Chetan summarized for Datanami, “At the end of the day, the Macrometa platform essentially deliver a full database, a 

 system like Kafka, and a complex event processing system 

 along with compute engine so you can build a real-time data application, completely distributed using us” via the 175-site GDN. 

With many tutorials, QuickStarts, and easy-to-follow example 

, you can get started with Macrometa in minutes. I mean you can actually build a streaming app in 10 mins or less! Start by requesting a 

Has Macrometa Cracked The Code For Delivering Data Value?

Wireless communication is so fundamental to our existence nowadays because it's at the very core of our digital lifestyles. The mobile phone has powerfully seduced and reprogrammed us socially, environmentally, and culturally. We all agree that the last ten years have unleashed a revolution of innovations via the mobile phone as it converged many disparate devices into one (a phone, a watch, a calendar, a contact book, a camera, a web browser, email, chat and so on) and then with the advent of faster cellular connectivity and programmability of the phone via general purpose developer tools, all kinds of amazing digital tools and services. With a few taps on the screen, we can summon literally anything – from infinite varieties of streaming video and gaming entertainment to food from our favorite restaurants, transportation to take us anywhere we want, reserve rooms and even houses in exotic locations around the world, and more. We can very literally perform almost all of the essential tasks required of us at home and at work on these magical devices. And all of this is only possible because the mobile phone is programmable using general-purpose tools and well-understood programming languages and frameworks. Basically, almost anyone with an interest in programming can build an app for the mobile phone if they are so inclined.

Now while the mobile phone has an obvious value to us because it's right there with us (all the time) as our most essential personal tool - the thing that makes it do these magical things is actually the cellular network that connects our magical mobile phone to the even more magical cloud where the backends for these apps run. Now developers love building apps on the cloud and developers love building apps on the mobile phone yet no one ever thinks about the network that connects the two unless it stops working. So the question that begs to be asked is: why is the network so tactical and why don't developers care about it as they care about the cloud or the mobile phone? I’ll tell you why - it's because you can't program the network to save your life if you needed to. The telecom network is literally the last important black box left in the technology world, that is opaque, unprogrammable, unfriendly, and resistant to change. While this has historically been justified as necessary to make it a critical infrastructure that can reliably deliver voice services to millions if not billions of users across a continent reliably and cheaply - the truth is that it's just not an infrastructure that can be leveraged to create new meaningful value unlike the cloud and the mobile phone.

 - the first of its kind cellular cloud that is literally breaking all the arcane and bureaucratic rules of the stodgy and slow telecommunications world by building an entirely new architecture for cellular networks that is cloud native. 

, founder, investor, wireless technologist, radical provocateur, entrepreneur, and now Vice President of the marketplace and app design at DISH Wireless explains it as “At DISH Wireless we are not building a wireless network, we are building a wireless cloud. One of the first killer apps for this wireless cloud is a 5G network”.

Put in more direct terms, what Brian is saying is that 5G is an application of a distributed hyper-cloud. What he is implying is that 5G is not the enabler of distributed hyper cloud, but rather that 5G (and all the Gs that come after) is the consequence of building a wireless infrastructure on cloud-native principles of agility, scaling from zero to infinity, programmability and making every part of the wireless network stack fungible via composability powered by SDKs and APIs. And it’s not just 5G that is the killer app but the ability to put apps and web services right on the radio so that they can programmatically access that radio network that makes the hyper-distributed wireless cloud the ultimate platform for app developers. And that's where our collaboration with Brian and DISH Wireless begins.

The DISH Wireless and Macrometa Partnership

In the relatively near future (early 2023 in a few months), Macrometa and DISH Wireless will provide a joint integration that enables app developers to not just build apps and APIs on 

 but also compose a wireless cloud on top of the DISH Wireless U.S. network on the fly, to deliver those apps directly from thousands of cell sites around the country - literally a millisecond or two away from almost everyone in the USA. 

This looks like a table on our poly model database that ingests or stores some data, then developers build a few APIs using 

 to ingest, transform, analyze and actuate the data. Once this is done, the network menu can be fired up, and you can select the locations around the country that you want to deploy the app on, configure the wireless stack by comprising a private LTE network from best of breed parts (via API or via our visual console), You can also configure network characteristics such as slicing and QoS, choosing a MEC instance size and provisioning a fleet of them right on the Radio Access Network that bind to your newly created private network. After this, click deploy - and your app lights up across the United States milliseconds later powering radical new apps and digital services, because now the entire cloud is not behind the telecom network but IS THE TELECOM NETWORK. 

Now everything is programmable - the backend edge cloud, the network, and the mobile device - and that is how all the amazing things we believe are in our digital futures will be realized. We at Macrometa refer to this as a state of “wait-lessness” - or no waiting. Imagine when you walk up to your rideshare uber car and the app and the network authenticate you instantly, unlocking the door, or when you check into a hotel and go up to your hotel room, your preferences for climate, lighting ambiance and even your credentials to various video streaming services and your corporate network secure access via a zero trust connection are all automatically configured as your door unlocks. You rock up to the gym and the machine pulls up your workout and automatically sets the weights and records your reps and pushes into your health app. The possibilities are endless and tantalizing.

Watch this fireside chant with Chetan and Brian for 

Ultimately, all this may just lead to this crazy thing the world is excited about - the metaverse - an augmented reality overlaid on top of our physical world that makes our world just a little more friendly, easy and pleasant to live in.

The future of the cloud is an edge, and the edge is already here on Macrometa and DISH Wireless. Come build on the cutting edge with Macrometa and friends. Request a 

The Rise Of The Wireless Cloud: How DISH Wireless And Macrometa’s Collaboration Will Make The 5G Wireless Network And Edge Powerful

If you’ve ever been frustrated at the age-old problem of coding products with unwieldy legacy tools while facing the relentless pressure of not going over time and over budget, we've got something for you.

Tangle is an intuitive open source platform that lets developers build enterprise applications in hours - with reusable components, drag and drop tools, secure architecture, and code extensibility. More importantly, it also provides developers a path to enhance quality further by contributing high quality, bespoke components - if and when needed. So, developers have complete control along with the powerful tools they need to save up to 90% of development time.

At Macrometa, we love everything user-friendly. So, we talked to Tangle about their low code/no code stream building capabilities and how that could apply to Macrometa’s powerful data processing platform.

With our combined expertise, what could developers and other data-obsessed teams do? Real-time collaboration experiences, real time data and management interfaces, analytics - quickly, simply, and without having to reinvent the wheel.

While we’re unveiling some exciting concepts and ideas today, the real deal’s on the way - and it will only be available at Macrometa.

Wayne Byrne, CEO at Tangle sat down with Michael Ridley, Head of Product Marketing at Macrometa to discuss low code at the edge. Take a peek into the future of edge development.

Curious what the combination of Macrometa GDN and Tangle’s low code technologies might look like? Watch the Macrometa+Tangle concept video to see new possibilities open up.

Stay tuned as we explore all the possibilities of the edge in partnership with Tangle.

Is Low Code The Future of Enterprise Edge?

, and today we're focusing on customer stories, more specifically, what people are building today on the GDN to solve real business problems that need a real-time edge to solve them.

The era of big data is now behind us, and the era of fast, global, and real-time data has begun. Coupled with data now becoming a phenomenon that polarizes with its aspects of privacy and exploitation, data is also becoming an area that nations and states are seeking to regulate, to protect themselves from big tech.

There are 3 business drivers behind the enterprise going beyond looking for tooling on historical data and looking at real time data tooling. In this post, we discuss how Macrometa’s customers are taking advantage of the 

Enterprises have realized that the hundreds of dashboards and thousands of reports that their tools produce every day are ineffective and will never enable better decisions, revenue or improve their cost efficiencies. For enterprises to be able to take data to the bank, they need data and their tools to enable better decisions by dealing with “what’s happening in the now” and not by analyzing what happened last month, quarter, season and year (except for compliance reasons).

But there are structural barriers to this adoption of real time tools. The tools that an enterprise puts to work on data today – like data warehouses, data lakes, and databases, are backward looking data infrastructure and are great at telling you what happened at noon last week Tuesday but have no ability to tell you what’s happening now. 

 can’t tell you what to show a visitor when they hit a website and is going to bounce 1.8 seconds later because they didn’t see anything interesting. It's also impossible for 

 to analyze the energy data coming from PG&E grid and correlate it with electric car data in order to help Tesla or Rivian figure out when it would be the least risky time to charge their vehicles during a heat wave in California without bringing the entire power grid down by oversubscribing.

Data monetization requires a tool chain that is “purpose built” for ingesting vast quantities of data in the “now” and making business decisions (analytics and AI) and actions (actuations that trigger automated workflows).

Resulticks - a SaaS disruptor in real-time Customer Data & omni-channel marketing

 (short for - Real Time Results from Clicks) is doing exactly this by building a massively concurrent omni-channel customer behavior analytics and attribution technology layer on Macrometa. Resulticks is one of the fastest growing SAAS companies in the Omni Channel Marketing & Customer Data Platform category. They came to Macrometa a year back, building a small real time web service for real time 

 (CEP) for a verticalized marketing web service for financial service customers that would make real time offers. Having had enormous success with Macrometa, they are now looking to build substantially new real time capabilities into their Customer Data Platform and data stack on Macrometa to gain a differentiation with real time as the lead value proposition. But more importantly, forward thinking SAAS disrupters like Resulticks are recognizing that data monetization needs new approaches to connect the dots of data into monetizable insight in 100 milliseconds or less and Macrometa’s real time CEP framework is probably the only such platform in the market to date. 

The team at Resulticks also has the hefty task of complying with a broad range of data handling and privacy regulations across different industries and geographies. Resulticks found that the Aerospike solution they deployed didn’t provide the speed and performance demanded by the marketers that use their real-time conversation platform worldwide. That’s when they turned to Macrometa.

“Our customer’s profitability and growth depend on our platform’s ability to go from data ingestion to action in milliseconds,” says Resulticks technical leadership and architect MS Kumar. “This requires a data platform that’s not just ultra fast but also versatile enough to handle complex data sets that combine streaming and structured data, enable real-time and batch queues, indexing, and data pipeline support as well as messaging in one integrated architecture.”

Business Driver #2 - Operational Analytics

Data lives fossilized as transactional data of record across hundreds if not thousands of silos in an enterprise. But data changes all the time in the real world. A website visitor bounces or becomes a prospect, a prospect becomes a customer with a click of a button that adds a payment to a subscription. And inside these silos, only the last action of a transaction and the state change between payment and order is captured, and all the metadata of what led to the transaction is lost forever. Transaction data fossilizes, frozen permanently in the amber resin of static transactional metadata and losing all relevance to the present and future because of the loss of operational metadata.

For real time operational analytics to be a reality, developers need tools and stacks for connecting operational data and metadata across boundaries (and silos). A technology platform that provides a streaming causal graph for tracking data and metadata changes, with a built-in complex event processor and real time pipeline that recalculates second and tertiary implications of this changing state as it happens. And finally, the technology platform must be able to put APIs on dynamic data to propagate it globally, across all the boundaries we discussed above. It should be able to make data flow between disparate silos and systems, across data centers and clouds, and across supply chains of information between teams, departments, divisions and partners.

AppSecEngineer - Delivering AppSec threat simulation and remediation training in real time

 provides labs, cyber-ranges and lessons on security focused areas like Application Security, Cloud Security, Kubernetes Security and more. Thousands of application security professionals use the AppSecEngineer platform everyday to train on threat scenarios in the constant battle to beat bad actors who may take advantage of bugs, exploits and vulnerabilities in these complex cloud environments.

To be able to provide a real time and highly realistic threat simulation and remediation lab, AppSecEngineer needs to ingest all kinds of data and enrich it for possible security threat events. Their platform captures high velocity logs and events, and transforms them into operational data that can be queried and investigated in milliseconds from the time the event is generated to the time it needs to be acted on. AppSecEngineer initially used a full AWS stack composed of 

 for log streaming, processing alerts and so on.

However, they quickly realized that this was neither developer-friendly nor scalable. 

, while a great logging service, also had some significant downsides for what they were trying to achieve. In the words of Abhay Bhargav, world renowned security expert and CEO of AppSecEngineer, “The native query interface of Cloudwatch was difficult to work with, especially for several thousand functions with multiple API Gateway instances that we need to query. We found this frustrating, to say the least." Because Cloudwatch comes with Log Limits, and the AppSecEngineer team wanted to retain logs for an indefinite period, they also dealt with major cost challenges of using CloudWatch. They also found annoyances like the fact that Dashboarding and Alerts were not easy to use with Cloudwatch. All of this resulted in becoming a pretty major bottleneck and constantly growing management overhead for them as interest and use of AppSecEnginer’s platform exploded.

So they decided to replace Cloudwatch with Macrometa, and stored the streamed logs as structured logs in JSON documents, and using 

, they added useful transformations in the form of additional context to the structured logs with service information, and our streaming graph engine to make graph relationships in real time on the dataset.

As Abhay puts it, this capability was not just revolutionary but radically simple, “All of this is radically easy with Macrometa because all we have to do is just create a query worker with specific parameters that we pass. This is an API that we invoke with the right parameters and the data is ingested into Macrometa and automatically becomes a graph. As a result we were able to get AppSecEngineer up and running very quickly on the 

 he says. “Firstly, we are able to capture events quickly with zero human intervention. Secondly, with the graphing capability we are able to model complex relationships and identify security anomalies and events very quickly on our stack.” He continues.

The result of using Macrometa instead of AWS Cloudwatch? “The much needed observability into our stack allows us to out-think attackers without ever having to “manage'' things” concludes Abhay.

Business Driver #3 – Context Sensitive Analytics

Enterprises collect data like they might go out of fashion any instant. While analytics on historical data points can be useful for many use cases, the real power of data analysis and actuation is unlocked by enriching data by synthesizing metadata about the relationships between historical data and events arriving in real time. This provides the ability to bring context to the data and enables contextual understanding and actuation based on similarities (is the data we are seeing arriving now seasonal or anomalistic), constraints (does the data arriving now differ from expectations of what previous data sets have informed us), paths (does the data show us different paths to the same point of point for a customer), and clustering around certain attributes (is the person in the chat window likely to buy a premium product or service based on the question they are asking).

The power of streaming graphs cannot be overemphasized for enabling this type of contextual enrichment of data – by processing events as they arrive and creating a context graph, enterprises can combine historical data and real time data to build powerful cognitive systems for real time decision support and actuation.

DevCycle - Re-imagining Feature Flags as a Real-Time Service

 has reimagined feature flag management for the real time era. Used by large international enterprise customers like Royal Bank of Canada (RBC), and fast growing SaaS companies like Humi and drone technology for the renewables energy sector Heliolytics - DevCycle wanted to differentiate itself by bringing ultra low latency capabilities deep into its core architecture. So they invented 

 - a new A lightning-fast, globally replicated edge storage engine that simplifies feature flagging in serverless environments enabling a new era of feature management with Edge Flags, Super Segments, Data Residency - all on Macrometa’s Global Data Network by building higher level SDKs and APIs for feature flags using the power of the 

.

After initially trying to implement the EdgeDB vision with 

 and being underwhelmed by the slow and performance, and high latencies to update and fetch data from these edge storage solutions, DevCycle turned to Macrometa and the Global Data Network and the results were in their own words “revolutionary”. Per Aaron Glazer, DevCycle’s highly energetic and visionary CEO “Data can now be saved as soon as it’s available and accessed by lightning-fast APIs to make near-instantaneous flagging decisions. All engineers need to leverage that data is a simple User ID – this level of simplicity is unheard of. Feature flags have never been real time or this fast and seamless until now”

While reading these fascinating scenarios of how our customers are solving some of the problems that are impossible to address with centralized public clouds, I hope that we have made a simple and powerful case that the edge can help solve unique problems in the way the cloud does not. Macrometa’s differentiated architecture as a hyper distributed Global Data Network and edge cloud for developers enables a very powerful and unique value proposition - take advantage of powerful business drivers like data monetization, operational analytics and contextual analytics to make your apps and APIs global and real time so that you can profit from the edge.

The future of the cloud is at the edge. And the Edge is already available on Macrometa. Come build on the cutting edge with Macrometa and friends like Resulticks, AppSecEngineer and DevCycle did.

What Macrometa Customers Are Building On The GDN

Learn how developers deliver flags instantly to anywhere in the world

 is a developer-first feature management platform that focuses on the full lifecycle of feature development - from code to production, validation, testing, and iterating. They’ve partnered with Macrometa to build 

 - an ultra low-latency global feature flags solution to deploy software faster and reduce release complexity.

Feature flags are a powerful tool that act as a toggle to switch features on and off. They are used to control the release of new features and functionality - especially to activate different features/functionality and experiences for different users.

Feature flags give development teams the ability to enable or disable features/functionality remotely without making experimental or embargoed features universally visible to users.

Feature flags decouple deployment from release - so engineers can quickly do their best work while product managers can get features out to users in the best way possible.

What is the DevCycle Edge Flags solution?

fastest way to deliver feature flags using data stored

 at the edge- are a key differentiator of the DevCycle experience.

Data for feature targeting isn’t always where you need it at the time you need it. With Edge Flags, developers only have to send the information once and DevCycle delivers flags instantly to anywhere in the world.

Developers use Edge Flags to make near-instantaneous flagging decisions using simple User IDs with real-time user data. The data is globally replicated at the edge and this lets them pass data once and deliver the same experience everywhere - across all devices, anywhere in the world. So developers don't have to set up expensive data pipelines to sync data across multiple platforms, applications, or microservices.

With edge-localized data synchronization, developers ensure that feature flags are updated in real-time, providing a better experience for customers and making it easier to test and iterate on new features. Features are turned on or turned off in core services at the server level (or on the website and mobile app) and that decision is made and implemented immediately across the entire system.

Real-time feature flags provide an additional layer of control by allowing development teams to change the behavior of a feature on the fly, without having to rewrite or redeploy code. This can be useful for things like A/B testing, error handling, or instantly rolling back a feature that isn't performing as expected.

In terms of customer experience, Edge Flags can be used to serve up customized landing pages for different sets of customers. For example, if a retail store knows they have two sets of customers - one that prefers to walk into the store and another that prefers to shop online - they can set this preference as a flag that toggles different landing pages with custom walk-in or delivery options. These preferences can be baked into a simple property such as User ID that - combined with targeting rules - sets up personalized user experiences.

 from DevCycle Product Manager Victor Vucicevich.

 to support the near-zero latency that Edge Flags mandates. Macrometa’s flexible, ultra low-latency data layer purpose-built for global, real-time, and event-driven use cases helps them solve the problem of making data instantly available everywhere without passing it around anywhere. 

With Edge DB providing real-time user data, development teams can deliver flags in an instant - without complicated data pipelines or tradeoffs between storage and latency.

Curious what it takes to build a game-changing real-time feature flags solution? Watch Cobi Druxerman (Co-Founder & Head of Product at Taplytics and DevCycle) in conversation with Justin Johnson (Head of Product at Macrometa) as they discuss the practicalities of implementing Edge Flags while also elevating the developer experience.

Interested in digging deeper and seeing Edge Flags in action? Head to the Edge Flags on Macrometa 

How DevCycle Built The World's Fastest Feature-Flag-As-A-Service On Macrometa

Authors: Taric Mirza, Founder, and CEO @ Arrive, and Alan Evans, Principal Technologist @ Macrometa

, is a lifelong gamer who’s made it his business to redefine the Esports and multiplayer gaming experience. When he hit severe lag playing League of Legends, he tried everything - changing ISPs, home routers, just name it - but nothing fixed it. Eventually, he decided to come up with his own solution to defeat lag for LoL and other games. His current favorite games are League of Legends and Apex Legends.

Today, Taric is in conversation with Alan Evans, Principal Technologist at Macrometa where he explains how Arrive conquers lag by finding the fastest network route. He also talks about how Macrometa’s 

 allows Arrive to manage its global infrastructure very easily and with very 

. Can you tell us a little bit about your company - Arrive - and your motivation for building it?

 Arrive is a startup company that is developing new products for Esports players to improve their experience playing games. The motivation behind starting Arrive was I hit severe lag playing League of Legends, and nothing I tried (changing ISPs, home routers, you name it) fixed it. Eventually, I switched from working on multiplayer network programming in my own small games to instead using some of those same techniques to come up with solutions for lag for other games. 

Our core product is an application for League of Legends players to improve their internet. We are working on supporting more games. I’m really excited about Arrive, as lag is a problem that impacts virtually every esports player at one point or another, and solving this would benefit players individually, as well as support the gaming ecosystem.

 - I was just thinking, though, some folks reading this might be asking what is “Esports? and how it is different from “normal” multiplayer gaming, and maybe describe what “Lag” is.

 - Sure! The main difference is Esports is more competitive in the sense you get an official Rank, much like chess players who are competitive get an “ELO” score that shows them numerically where they stand in relation to other players. For Esports, while having casual fun is of course still the main reason people play, there is a real sense of “serious competition” (that in turn, makes it even more fun!). At the highest levels, players have dedicated gaming teams they are on that receive significant corporate funding. Much like professional sports teams, at this level, players can get traded to other teams. The prize pool for some tournaments can exceed $1 million, and some of the top players earn that much or more from their teams as well. 

 - Ok, so Esports is more competitive, got it. Plus, the gamers get sponsorship deals, I saw, so there is lots of advertising involved as well, I guess?

 - Esports is part of the competitive video game online arena where players play the game competitively. These are games where players play against other players in each game and get assigned a rank based on how well they perform and if they win. The most successful esports games have tournaments that are watched by millions of people worldwide.

 - love watching the professionals playing LoL; there is so much happening on-screen. You can really see why lag can be a real problem.

 - Yes, and while the official tournaments hosted in person are lag-free (as the computers are right next to each other and directly connected), many LoL tournaments are online, where players sometimes play hundreds of miles away from each other. Lag is a huge problem in these cases, and Arrive would like to play a role in working with some of these tournament operations to help fix these issues.

Why is Traffic Acceleration critical for Esports and Multiplayer Game traffic?

 - Ah, I would have loved to have been an Esports player. When I first got into online multiplayer gaming, it was on the Sony PlayStation, and a game called SOCOM Navy Seals. Super competitive, and if you saw people lagging across the screen, it was a real nightmare as you just didn't know what was going to happen. Bad lag can really ruin multiplayer gaming, no matter how competitive the game is.

 - Yes, Internet lag is one of the top problems that video game players, particularly esports players, run into. Multiplayer Game traffic Acceleration, which Arrive is building with help from Macrometa on the 

, is able to reduce or eliminate lag for most gamers.

 - Taric, do you think you could describe or maybe explain what the Arrive solution does?

 - The solution includes client software that the player installs on their gaming computer that routes the game traffic onto the Arrive network. It is simple to operate as the gamer – you basically just turn it on and off as needed, but behind the scenes, the solution is scouting and utilizing the best internet routes available on the Arrive network to enhance the gaming experience.

 - Ah, since you mentioned Macrometa, why are you using the Global Data Network (GDN)?

 - To power Arrive’s product that accelerates network traffic, Arrive manages a fleet of servers in a variety of data centers spread across the world. Macrometa’s GDN allows Arrive to manage this global infrastructure very easily and with very low latency.

 - Nice. Yes, our Global Data Network is available in almost 200 locations around the world, providing a P90 sub-50ms RTT to any connected device globally. 

How do you leverage Macrometa GDN to Conquer Lag and Accelerate Traffic?

 - Do you think you could go into more detail on how Macrometa is being used under the hood for Arrive?

 - Sure, we’ve found Macrometa to be a game-changing (pardon the pun) :) platform that has allowed us to develop the end-to-end Arrive solution much more efficiently than would have been possible without Macrometa. 

Macrometa is our primary - and only database - and PaaS. It handles 100% of our persistent (and mostly real-time) data needs which is pretty incredible for a single technology. We use Macrometa GDN for many areas but the basic approach is that we use Macrometa to contain both our primary event and transaction data, in terms of being a “single source of truth” for data, but also operational data such as aggregations, analysis, and reporting.

In addition, we use Macrometa for our User Management system, handling everything from registration, authentication, and logins to what features the users have enabled.

Macrometa also powers our server management and resource allocation technology. This is the system that figures out which of our servers or network paths have sufficient capacity to serve more players who want optimization. There is quite a bit of complexity in keeping track of all this, and Macrometa makes it easy.

 data too. This is just another aspect of how Macrometa shines – in the old days, you’d have to research and integrate with a separate caching system requiring its own command language/API, integration between the databases, and pricing. With Macrometa, we get an easy-to-use, cost-effective, high-performance Cache system built right in. 

And finally, we use Macrometa for our internal reporting systems, too. Things like aggregating data and generating reports with it are easy with Macrometa. If we want to know how many players used Arrive today, Macrometa by itself answers that question. So yeah - we use Macrometa across the board as the platform that enables us to focus on building the world’s gamer service for multiplayer gaming - it lets us focus on what we do best - making games awesome for gamers by eliminating lag, improving jitter and latency by leveraging the edge in ways that are impossible to do in the cloud.

 - Wow, impressive, that’s quite a bit! It’s interesting to see how much you are using 

. I’m particularly curious about what made you choose Macrometa to be your primary platform as opposed to one system as a part of many others.

 - The biggest gain of leveraging Macrometa handling everything as opposed to a patchwork of various systems is it drastically reduces complexity. That, in turn, leads to many significant benefits:

First of all, we don’t need to pay for 3rd party ETL or spend a considerable amount of time developing it ourselves. This is all done automatically by Macrometa for us. This saves us quite a bit in both financial and labor costs.

Secondly, a few systems make our application much easier to operate. Things like configuration, software releases, testing, and general day-to-day development are all more straightforward. Even onboarding new developers is much faster when they can learn just a single system instead of 5-20 that all have to fit together.

And last but not the least, we also benefit from reduced risk and tighter security. Generally speaking, the more complex an architecture, the greater risk there is for an outage or security incident. It is more challenging to secure and operate a whole patchwork of various databases, ETL, and reporting systems, all with various synchronization processes, authentication keys, etc., than a single system.

 - Excellent! So, could you talk about what challenges Macrometa has helped Arrive overcome?

 - One of our greatest challenges is building a system that can handle users globally, with high performance, and that can scale very well as we continue to grow.

One of the great things about Macrometa is you can come up to speed incredibly fast, which is critical for us as a startup company, as getting a great product out as soon as possible is essential.

At the same time, we wanted to find a technology that would handle users around the world and scale. Macrometa handles all this for us seamlessly. With other technologies, this generally takes very expensive projects consisting of top consultants to help get all the geographic distribution, redundancy, security, and scaling aspects. What previously might take 6-12 months of work, best case, you now get on day 1 with Macrometa right out of the gate!

 - Thanks Taric. So, where do you see this going? I mean, what do you think the next steps are for Arrive?



 - For Arrive, on the product development side, we want to support many more popular Esports games. We want to expand to support players in more cities and countries around the world. We plan to improve the algorithms we use for internet traffic routing to improve gamers’ experience further. On the distribution side of things, we will partner with enterprises like ISPs to help get our product out to more players.

 - Can Macrometa help in any of these areas?

 - Macrometa can help with all these areas, actually!

For game expansion, Macrometa will play a critical role in this, as many of the other games we want to support require considerable data processing. Macrometa is ideal for this. We will evaluate using Macrometa’s Streaming technologies for some aspects.

Same story with the routing algorithms. Our next iteration uses more involved logic, and Macrometa’s data processing capabilities will make this part much easier. The 

 might be particularly powerful in enabling some of the features we want to add.

Macrometa will help not just on the tech and data side but also on the business side in terms of helping develop and bring a product to ISPs using the Macrometa GDN PoPs inside the ISP networks.

 - Hey Taric, this has been great. Have really enjoyed our chat, as usual, and I'm sure our readers have as well. Have you any closing remarks or ideas you would like to share? and where can our readers find out more about Arrive?

I’d first want to say I’ve really enjoyed our discussions and would extend that to some others at Macrometa. When I first reached out to Macrometa with a list (some might say a book!) of questions on how the technology works, Justin Johnson was amazing at answering everything. I also had the pleasure of meeting Macrometa’s CEO, Chetan, in person in Dallas earlier this year. Not only was it a productive conversation, but as a human being, Chetan is amazing and inspiring.

To learn more about Arrive, please check out our website, 

, and I’m always happy to answer any questions (or play LoL or Apex)!

We’re excited about the future, not only fixing lag for video game players as we’re doing today but expanding into providing valuable tools for players in other aspects of their gaming experience and helping support the greater Esports ecosystem, and extending our network optimization technology into all sorts of other areas where real-time performance is key.

If any of our readers would like to learn more about Macrometa GDN, request a 30 day trial. 

How Arrive Conquers Lag And Accelerates Esports Traffic With Macrometa Global Data Network

This is a guest post from Abhay Bhargav, the Founder and Chief Research Officer of 

“Attackers think in Graphs, but Security thinks in Lists” is a famous saying amongst security practitioners like myself. And this is very true.

 - who popularized this saying - pointed out that defenders or security teams think in terms of asset lists while attackers think of the relationships between different assets. They breach a system using one (or more) of the assets and find multiple paths to the highest value asset by navigating the “graph” of interlinked vulnerable assets. 

On the modern web, attackers look to identify and exploit vulnerabilities. Once they compromise an application(s), they pivot into internal applications and connected cloud services through chained vulnerabilities and exploits. These attack sequences lead to the eventual outcome that the attackers desire-, which could be anything from stealing sensitive information to deleting mission-critical datasets and disrupting operations.

Attackers honestly think in graphs. Their ability to form relationships between connected assets gets them the ultimate prize.

Security teams need to pivot from list-thinking to graph-thinking to defend their systems successfully. This means they must visualize the entire network as a series of security dependencies and implement controls to reduce the attack surface.

That’s precisely what we set out to do at 

We’re a leading EdTech Platform for Application Security. We provide labs, cyber ranges, and lessons on security-focused areas like Application Security, Cloud Security, Kubernetes Security, and more. As a result, thousands of students, from job seekers to experienced professionals, use our platform daily to upskill themselves and improve their job prospects.

Being a security training platform, we can’t help but take security seriously. One significant aspect of “taking security seriously” is to capture and enrich logs and possible security events. The idea is to capture logs and events we ingest and further transform them into data we can query and investigate as the need arises.

Our platform is serverless from the get-go as we were “born in the cloud”. We use 

 for object storage. Our application and user base have grown exponentially over the past year that we’ve been in business. 

 for log streaming, alerts, etc. However, we quickly realized this could have been more user-friendly and scalable. Cloudwatch is an excellent logging service, but it also has some significant downsides for our use case: 

Its native query interface is challenging to work with, especially for several thousand functions with multiple API Gateway instances you need to query. We found this frustrating, to say the least.

Cloudwatch comes with Log Limits. And that is fine. But, we wanted to retain logs for an indefinite period. This proved to be expensive with Cloudwatch.

Dashboarding and Alerts are not the easiest with Cloudwatch. It becomes a significant bottleneck and management overhead to manage all of this.

Looking at the downsides of Cloudwatch, we decided to move to 

. As the Chief Technologist, I was already a fan of Macrometa because of its ability to ingest multiple data types, from document to key: value to graph, etc.

In addition, since Macrometa is serverless, it makes management overhead “not our problem,” - which we use as a basis to evaluate every solution we use. We have a small team, so “managing” a lot of resources is a NO-GO. In addition, I was interested in leveraging Macrometa’s 

Archive