Announcing Stream Workers: Complex Event Processing And Streaming Analytics At The Edge
Complex event processing is a tricky topic, made simple by Macrometa’s Global Data Network. Macrometa recently held a webinar giving an overview of CEP and the distinction between CEP and streaming analytics.
CEP allows you to work with one or more data streams, optionally combined with data-at-rest, to infer “complex events”. For example, you might have streaming financial trade data, and by joining that with other data about macroeconomic conditions, you could infer a “market correction” event.
Meet Macrometa Stream Workers
Stream Workers provides typical streaming analytics capabilities, including joining streams, aggregations, counts, and filtering as well as sophisticated high-level primitives for complex event processing such as pattern recognition, conditional logic, and flow control. The Stream Workers engine runs within the Global Data Mesh, allowing you to incorporate operational data-at-rest with streams of data from both Macrometa’s streaming collections and other event streams, such as from Apache Kafka topics. Stream Workers can write to data stores within the Global Data Mesh or stream results to other streaming frameworks like Kafka. In addition, Stream Worker's applications can be queried directly using a REST API. Since Stream Workers is tightly integrated with the Global Data Mesh, there is no need to transfer data from your operational data store to the CEP engine. And they are, ultimately, making large amounts of reference data available instantly for sophisticated predictive modeling at very low latency.
What can you build with Stream Workers?
Stream Workers can write self-contained microservices and traditional streaming analytics applications. Using Stream Workers, you can create REST APIs, create microservices, and serve the logic of an application backend. To demonstrate this, we’ll show our cryptocurrency trading demo below. Stream Workers can solve so many problems, but here are some simple use cases:
Transform data from one format to another, for example, from XML to JSON, and clean and filter your data:
- Extract, transform, and load (ETL) data from log files and scrape HTTP endpoints.
- Obfuscate sensitive information by filtering it from messages, logs, or other data streams that may include Personally Identifiable Information (PII) or Personal Health Information (PHI).
- Correlate data by joining multiple streams to create an aggregate stream.
Complex Event Processing
Derive insights by identifying event patterns in data streams:
- Summarize data with time windows and incremental aggregations
- Trigger actions based on streaming data, either as a single service request or a complex enterprise integration flow.
- Build and refine predictive models for machine learning
Machine Learning Model Serving
Build, refine, and serve machine learning models:
- Make real-time offers to customers who are visiting an eCommerce site based on browsing or shopping history
- Threat detection in cybersecurity applications
- Predictive maintenance use cases
Stream Workers Demo: High-Frequency Crypto Trading App
In this demo application, we build a crypto arbitrage trading bot to trade Bitcoin locally in US, Germany, and Japan exchanges.
An exchange arbitrage is a trading strategy based on the differences between the price of the cryptocurrency at different exchanges. Generally, these opportunities open up if there is a price discrepancy, and the contrast accumulates over time to finally become a significant amount, generally 1–3%. However, these arbitrage opportunities typically only exist for seconds, so an arbitrage trader must constantly search for the best opportunities and implement them immediately.
Implementing arbitrage strategies in the crypto market takes work and involves several technical complexities. Besides having deposits in multiple major exchanges, on the technical side, it consists of the ability to monitor arbitrage opportunities simultaneously, making trades locally at each exchange with minimum delay and timely visibility and interactions between the trading agents distributed globally. The challenge requires big dollars and, until now, a team with the technical expertise to build a distributed infrastructure.
To build the demo, we used the following:
- Macrometa Global Data Mesh collections, both streaming and data-at-rest, and Macrometa Edge Compute’s Stream Workers CEP engine.
- We use a document store to store trade data and share all trade-related metadata and state between the decentralized, stateless bots and components.
- We use geo-distributed streams for publishing real-time price data from each exchange. Each bot subscribes to the stream and decides on whether to buy or sell.
- Lastly, use real-time updates from the datastore to notify the clients/bots/subscribers of changes to the database so they can modify their trading strategy.
The Stream Workers Developer Experience
We provide native sources and sinks to connect to Kafka, Google Pub/Sub, MQTT, and S3.
The Stream Workers code editor
After you’ve written your code and are ready to save the Stream Worker, click save, and pick the regions to deploy your Stream Worker.
Create Stream Worker
When you’re ready to go live, click Publish. You don’t need to manage any software images or configuration; we get your app up and running in a few seconds. Macrometa’s intelligent traffic routing means every time a user or client sends data to the GDN, and it’s routed to the region closest to them. That means your applications will be FAST.
We provide several sample Stream Workers samples in the web console to get started quickly.
We think Stream Workers is ushering in the next era of developing applications at the edge, but don’t just take our word for it. Instead, read through our documentation to see all the available APIs, and request a trial to try them out. We can’t wait to see what you build.