POW! The challenges of log analytics
Real-time log analysis is complex and can require a huge amount of resources and tools. Having visibility of all these tools is an added challenge. At Macrometa, we give developers superpowers because we offer data storage and processing in one friendly interface. Tired of Kinesis or Splunk, or just want to explore new solutions in real-time log analytics? Macrometa and Fastly to the rescue!
Let’s talk about the technical architecture used to build a dashboard for real-time log analytics. This log analytics dashboard is part of a demo application called Edgely+ that Macrometa recently developed with our partner Fastly.
The volume of logs produced by a streaming service like Netflix or Disney+ would challenge anyone tasked with creating and supporting an analytics dashboard. They may end up just hoping a superhero will swoop in and save the day. Not all heroes wear capes and Macrometa is here to save the day!
Macrometa’s platform combines a performant database with stream processing capabilities for Complex Event Processing along with a globally-distributed architecture. This allows us to process and aggregate large volumes of log data in a single bound from across the internet with very low latency and a consistent database.
This blog will describe the architecture of the log analytics application and explain the various components of the solution. We'll also give you access to the GitHub repo so you can try it our for yourself!
Macrometa Fastly Log Demo Application
We have published the example log processing application code on GitHub and the repo’s README includes step-by-step instructions on how to install, configure, and run the demo both locally and on Fastly’s [email protected] infrastructure. The application consists of Stream Workers, Query Workers, Macrometa collections, a log capture service, and a web dashboard written in React.
Stream Workers is Macrometa’s stream processing framework and can be used for data processing tasks including transformation, enrichment, and aggregation.
In our Fastly log processing application, we use three stream workers. One sends HTTP requests to the Fastly service every three seconds to generate logs to analyze and the other two are responsible for reading and aggregating those Fastly web logs.
Stream Worker Fastly-log-generator
This Stream Worker uses triggers to make three HTTP requests to the Fastly service every three seconds.
Stream Worker fastly-http-request-worker
This Stream worker reads Fastly logs from the `logs` collection and performs reordering for out of order logs. Based on the log timestamp, it determines the 10 second window and adds that window timestamp as an additional field.
Stream Worker Fastly-http-request-stats-1m-worker
This Stream worker reads logs from the `fastly-intermediate-stream` collection and performs windowing aggregation. It performs aggregation for response_status, url, response_body_size, and latency.
Query Workers allows developers to store parameterized Macrometa queries within the service and access them via a REST endpoint. Developers consuming data from Macrometa can thus access the data through this REST endpoint without having to provide the actual query, allowing a separation of business logic from presentation code.
Our log monitoring application uses the following Query Workers:
The dashboard in the React application is populated with the results of these queries.
Log Capture Service
Fastly pushes web logs to a Python application that inserts them into a Macrometa collection using the Macrometa Python driver. Once in the Macrometa stream collection, Stream Workers process and aggregate the logs.
Stay tuned for the next adventure…
We'll go into more details on Stream Workers and Query Workers in future blogs. Sign up for a forever free dev account and see everything you can do in one platform.
- The Guide to Event Stream Processing
- Advantages of pub/sub
- How to make your own Twitter in 5 days
- Fastly partners with Macrometa for Partner Ecosystem