Transport protocol

Streams relies on server-sent events (SSE) for its real-time data push capability.

Connection heartbeat

In certain cases, some legacy network infrastructure may drop HTTP connections after a short timeout. To protect against such behaviors, Streams sends the client a comment line (starting with a ':' character) every 10 seconds. This comment line is ignored by the SSE client and has no effect other than a very limited network consumption.

When no change is detected by Streams, the subscribers gets those heartbeats repeatedly until an event is finally sent.

Compression

SSE flow can be compressed on demand using gzip or deflate methods. The following is an example of how to use the X-Sd-Compress header:

$ curl "https://streamdata.motwin.net/https://stockmarket.streamdata.io/prices?X-Sd-Token={{YOUR_APP_TOKEN}}" -H "Accept-Encoding: gzip, deflate" -H "X-Sd-Compress: true" --compress

If this header is not provided, the default behavior is not to compress the data flow.

Response format

Streams responds with JSON and JSON Patch.

After reading this topic, you should understand the result of your previous cURL commands.

How SSE is used

When you connect to an SSE server you receive an HTTP 200 OK code just like a regular successful HTTP connection. However, the connection remains alive and everything continues to happen afterwards, including errors (for example, authentication errors, bad requests). As long as the client or server does not end the connection, it remains alive.

SSE is a text-based protocol. The following is an example of the server’s response once the connection is successfully established (the metadata, headers, status codes are omitted intentionally):

id: 00ae73f5-5349-40c4-91b6-2e58a36b5365 
event: snapshot 
data : [{ 
  "id": "acb07740-6b39-4e8b-a81a-0b678516088c",
  "title": "94% of Banking Firms Can’t Deliver on ‘Personalization Promise’",
  "date": "2018-09-10-T10:13:32",
  "abstract": "One of the strongest differentiators ..."
},{
  "id": "0c5b5894-a211-47de-87a8-c7fa3ce3dfa2",
  "title": "Would you trust your salary to start-up",
  "date": "2018-09-10-T09:59:32",
  "abstract": "We take a closer look at how safe..."
}]

Line number key:

  1. id identifies the message.
  2. event is the name of the event that can be either snapshot, patch or error.
  3. data is the body of the message, always in JSON, in this case a JSON array.

These three fields are always present and represent a single message also called an event.

The snapshot event is only emitted once, after the connection is successfully established (unless you explicitly ask for snapshots only).

An error event ends the connection in most cases. See Error handling for more details.

JSON Patch

After the initial snapshot has been emitted, it is followed by patch events when Streams detects a change in the published content. The following is an example of a patch sent by Streams.

id: 37740aa3-3629-41c4-9a7f-24a1347383eb
event: patch 
data: [{"op":"remove","path":"/1"}]

A patch is a JSON document that gives you the difference between two JSON documents. It is represented by an array of operations to apply to the previous version of the document.

An operation is made of two or three fields:

  • op defines the type of operation (e.g. add, remove, replace)
  • path defines where the operation applies in the document (JSON Pointer)
  • value (optional) defines the value to apply: a raw JSON literal, object or array
Operation type Has a value
add yes
replace yes
remove no

In the preceding patch example, there is only one operation, the patch indicates that the second row has to be removed. A patch can be applied to an existing document to alter it (several libraries are available to do that). The use of JSON patch format enables Streams to save bandwidth by pushing only the differences between two versions of published content.

See Github repos for examples in different languages to help you with the connection management, SSE payload decoding and patch processing.

Related Links