Skip to main content

tokenize (Stream Processor)

Tokenize the list and return each key, value as new attributes in events


list:tokenize(<OBJECT> list)
list:tokenize(<OBJECT> list, <OBJECT> ...)

Query Parameters

NameDescriptionDefault ValuePossible Data TypesOptionalDynamic
listArray list which needs to be tokenizedOBJECTNoYes

Extra Return Attributes

NameDescriptionPossible Types
indexIndex of an entry consisted in the listINT
valueValue of an entry consisted in the listOBJECT

Example 1


The list:tokenize(customList) function takes a list named customList, and for each element in the list, it generates an event with the corresponding value. In this example, if customList contains elements (gdn, IBM, XYZ), the function will generate three events with value attributes gdn, IBM, and XYZ respectively.

Example 2

CREATE STREAM ListStream (customList OBJECT);
CREATE SINK STREAM OutputStream (index INT, value OBJECT);

@info(name = 'TokenizeList')
INSERT INTO OutputStream
SELECT index, value
FROM ListStream#list:tokenize(customList);

In this corrected stream worker, the list:tokenize function is invoked as part of the FROM clause, directly on the ListStream. It generates an event for each element in customList, with each event containing the index and the value of the corresponding list element. These events are inserted into the OutputStream. The OutputStream will then contain the index and value of each element in customList.