Skip to main content

Functions

This section lists all the stream worker functions provided by Macrometa GDN and explains how they work.

Functions enhance Macrometa Stream QL, the language used to write stream workers, by seamlessly incorporating additional capabilities. Macrometa Stream QL provides various in-built functions to access and manage event data according to our requirements. Functions can accept zero or more parameters, perform actions and return the result.

Functions allow you to:

  • Use different data sources.
  • Use different sinks, such as GDN Streams, Kafka, and ActiveMQ.
  • Appropriate format mapping, such as JSON or CSV.
  • Different kinds of processing, such as math, string, statistics, and geospatial.
  • Data aggregation.
  • Write logical expressions.

Core Functions

FunctionDescription
andReturns the results of AND operation for all the events.
avgCalculates the average for all the events.
batchA window that holds an incoming events batch.
castConverts the first parameter according to the cast.to parameter.
coalesceReturns the value of the first input parameter that is not null, and all input parameters have to be on the same type.
convertConverts the first input parameter according to the convertedTo parameter.
countReturns the count of all the events.
createSetIncludes the given input parameter in a java.util.HashSet and returns the set.
cronThis window outputs the arriving events as and when they arrive, and resets (expires) the window periodically based on the given cron expression.
currentTimeMillisReturns the current timestamp of stream processor application in milliseconds.
defaultChecks if the attribute parameter is null and if so returns the value of the default parameter.
delayA delay window holds events for a specific time period that is regarded as a delay period before processing them.
distinctCountThis returns the count of distinct occurrences for a given arg.
eventTimestampReturns the timestamp of the processed event.
externalTimeA sliding time window based on external time.
externalTimeBatchA batch (tumbling) time window based on external time.
ifThenElseEvaluates the condition parameter and returns value of the if.expression.
instanceOfBooleanChecks whether the parameter is an instance of Boolean or not.
instanceOfDoubleChecks whether the parameter is an instance of Double or not.
instanceOfFloatChecks if the parameter is an instance of Float or not.
instanceOfIntegerChecks whether the parameter is an instance of Integer or not.
instanceOfLongChecks whether the parameter is an instance of Long or not.
instanceOfStringChecks whether the parameter is an instance of String or not.
lengthA sliding length window that holds the last window.length events at a given time, and gets updated for each arrival and expiration.
lengthBatchA batch (tumbling) length window that holds and process a number of events as specified in the window.length.
logLogs the message on the given priority with or without the processed event.
maxReturns the maximum value for all the events.
maxForeverThis is the attribute aggregator to store the maximum value for a given attribute.
maximumReturns the maximum value of the input parameters.
minReturns the minimum value for all the events.
minForeverThis is the attribute aggregator to store the minimum value for a given attribute throughout the lifetime of the query regardless of any windows.
minimumReturns the minimum value of the input parameters.
orReturns the results of OR operation for all the events.
pol2CartThe pol2Cart function calculating the cartesian coordinates x & y for the given theta.
sessionHolds events that belong to a session.
sizeOfSetReturns the size of an object of type java.util.Set.
sortThis window holds a batch of events that equal the number specified as the windowLength and sorts them in the given order.
stdDevReturns the calculated standard deviation for all the events.
sumReturns the sum for all the events.
timeA sliding time window that holds events that arrived during the last windowTime period at a given time.
timeBatchA batch (tumbling) time window that holds and process events that arrive during window.time period as a batch.
timeLengthA sliding time window that, at a given time holds the last window.length events.
unionSetUnion multiple sets.
uuidGenerates a UUID (Universally Unique Identifier).

Execution Function Types

FunctionsDescription
ContextThis function provides useful environment properties such as current region where the Stream App is running.
MapThis function provides capability to generate and manipulate map (key-value) data objects.
JSONThis function provides capability to retrieve, insert, and modify JSON elements.
ListThis function provides capability to generate and manipulate list data objects.
MathThis function provides useful mathematical functions such as power, round, random, cos, log, etc.
RegexThis function provides basic RegEx execution capabilities such as find, match, etc.
ReorderThis function orders out-of-order event arrivals using algorithms such as K-Slack and alpha K-Stack.
TimeThis function provides time related functionality such as getting current time, current date, manipulating/formatting dates, etc.
Streaming MLThis function provides streaming machine learning (clustering, classification and regression) on event streams.
AnonymizerThe Anonymizer function provides a function for anonymizing various data types. This function returns a fake value for anonymizing which matches the original data. For example, an email would be replaced with a fake email.
CacheThe cache function provides a persistent cache per tenant.
Geo SpatialThis function provides geo data related functionality such as checking whether a given geo coordinate is within a predefined geo-fence, finding distance between 2 geo coordinates etc.
SentimentThis function performs sentiment analysis using AFINN Wordlist-based approach.

IO Function Types

FunctionDescription
Google Pub-SubThis an function that receives and publishes events from/to Google Pub/Sub.
HTTP and HTTPSThe http function receives and publishes events via HTTP and HTTPS transports, calls external services, and serves incoming requests and provide synchronous responses.
Apache Kafka (beta)This function receives and publishes events from/to Apache Kafka.
MQTTThis function receives and publishes events to and from MQTT.
S3This function allows to publish events to Amazon AWS S3 buckets.
SSEThis function receives and publishes events from SSE server.

Format Mappers Function Types

FunctionDescription
JSONThis function converts JSON messages to/from stream processor events.
CSVThis function converts messages with CSV format to/from stream processor events.
Key-ValueThis function converts events having Key-Value maps to/from stream processor events.
TextThis function that converts text messages to/from stream processor events.

Example

This stream application with the name TestFunctions creates a stream named FooStream. This configuration for SOURCE FooStream performs input mapping using JSON Format Mappers function. For a single event, the input is required to be in following format:

{    
"event":{        
"symbol":"GDN",        
"price":"55.6",        
"volume":"100"    
}
}

On receiving the event, a query is executed that parses string data types using execution function (math) into required formats. After that, the data gets inserted into Kafka SINK using Apache Kafka IO function.

    @App:name('TestFunctions') 

CREATE SOURCE FooStream WITH (type='stream', topic='stock', map.type='json') (symbol string, price string, volume string);

@info(name = 'query1')

CREATE SINK BarStream WITH (type='kafka', topic='topic_with_partitions', partition.no='0', bootstrap.servers='localhost:9092', map.type='json') (symbol string, price double, volume long);

insert into BarStream
select symbol, math:parseDouble(price), math:parseLong(volume)
from FooStream;