Skip to main content

Streams

Streams are flows of data in GDN to capture data in motion. Messages are sent via streams by publishers to consumers who then do something with the message. Streams can be created with client SDKs, REST APIs, CLI commands, or the web console.

Streams Workflow

Streams in GDN is built on the publish-subscribe pattern, aka pub-sub. In this pattern, producers publish messages to streams. Consumers can then subscribe to those streams, process incoming messages, and send an acknowledgement when processing is complete.

producer/publisher --> stream --> subscription --> consumer

For more information on these topics, refer to Stream Basics.

Producers/Publishers

Messages from producers (publishers of messages) are only stored once on a stream, and can be consumed as many times as necessary by consumers. The stream is the source of truth for consumption. Although messages are only stored once on the stream, there can be different ways of consuming these messages.

Subscriptions

Once a subscription has been created, GDN streams retain all messages even if the consumer disconnects from the server. Retained messages are only discarded when a consumer acknowledges that they've been successfully processed.

Streams

A stream is a named channel for sending messages. Each stream is backed by a distributed append-only log and can be local (at one edge location only) or global (across all edge locations in the fabric).

Consumers

Consumers are grouped together for consuming messages. Each group of consumers is a subscription on a stream. Each consumer group can have its own way of consuming the messages: exclusively, shared, or failover.

Benefits

Streams provide:

  • Seamless geo-replication of messages across regions
  • Very low publish and end-to-end latency
  • Seamless scalability to over a million topics
  • Multiple subscription modes (exclusive, shared, and failover) for streams
  • Guaranteed message delivery with persistent message storage

Stream Names

As in other pub-sub systems, streams in GDN are named channels for transmitting messages from producers to consumers. Stream names are URLs that have a well-defined structure:

persistent://tenant/fabric/stream-name
Stream name componentDescription
persistentThis identifies the type of stream. GDN currently supports only persistent streams. With persistent streams, all messages are durably persisted on disk (that means on multiple disks).
tenantThe stream tenant within the instance. Tenants are essential to multi-tenancy in GDN
geofabricThe administrative unit of the stream, which acts as a grouping and geo-fencing mechanism for related streams. Stream configuration is performed at the fabric level. Each tenant can have multiple fabrics.
stream-nameThe final part of the name. Stream names are freeform.

Streams and Fabrics

A fabric is a geo-fenced grouping within a tenant. A tenant can create multiple fabrics. For more information about fabrics, refer to Fabrics.

For example, a tenant with different applications can create a separate fabric for each application. A fabric allows the application to create and manage a hierarchy of streams. The stream my-tenant/app1 is a fabric for the application app1 for my-tenant. You can create any number of streams under the fabric.

note

Every collection within a fabric is also a stream with the same name.