Skip to main content

tokenize (Stream Processor)

This function splits the input string into tokens using a given regular expression and returns the split tokens.


str:tokenize(<STRING> input.string, <STRING> regex)
str:tokenize(<STRING> input.string, <STRING> regex, <BOOL> distinct)

Query Parameters

NameDescriptionDefault ValuePossible Data TypesOptionalDynamic
input.stringThe input string which needs to be split.STRINGNoYes
regexThe string value which is used to tokenize the input.string.STRINGNoYes
distinctThis flag is used to return only distinct values.falseBOOLYesYes

Extra Return Attributes

NameDescriptionPossible Types
tokenThe attribute which contains a single token.STRING

Example 1

CREATE STREAM inputStream (str string);

@info(name = 'query1')
INSERT INTO outputStream
SELECT token
FROM inputStream#str:tokenize(str, ',');

The query1 processes events from the inputStream and uses the #str:tokenize(str, ',') function to split the str attribute based on the specified delimiter, which is a comma (,). The query outputs the resulting token attribute values as separate events to the outputStream.

For example, if the input string is "Android,Windows8,iOS", the query will produce three events in the outputStream with token attribute values of "Android", "Windows8", and "iOS".