HTTP Streaming: One Response, Many Chunks of Data

A response can be produced over time

The model can look like this:

client -> request

client <- headers
client <- bytes
client <- bytes
client <- bytes
client <- EOF

The most important change:

response received

is not a single moment.

The headers can be received while the body is still being produced.

Fetch exposes a stream

const response = await fetch("/events");

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { value, done } = await reader.read();

  if (done) break;

  const chunk = decoder.decode(value, {
    stream: true
  });

  process(chunk);
}

response.body is a ReadableStream.

That lets the application consume data without waiting for the response to finish.

A chunk is not a message

This is one of the most important things about streaming.

If the server logically sent:

{"id":42,"status":"completed"}

the client may receive the bytes as:

{"id":42,

and then:

"status":"completed"}

or several logical messages in a single read.

Transport chunk boundaries are not application record boundaries.

That is why we need framing.

NDJSON

One simple format is Newline Delimited JSON:

{"id":1,"status":"processing"}
{"id":2,"status":"processing"}
{"id":1,"status":"completed"}

Every record ends with a newline.

The client buffers data until it finds the separator:

bytes
  |
  v
buffer
  |
  v
newline found
  |
  v
parse one JSON object

The alternatives are length-prefix framing, a custom binary format or another application protocol.

HTTP streaming gives bytes.

The meaning of those bytes is up to us.

Streaming does not mean HTTP/1.1 chunked encoding

Historically, HTTP/1.1 streaming was often associated with:

Transfer-Encoding: chunked

But that is a detail of HTTP/1.1 framing.

In HTTP/2 and HTTP/3 the body can also be delivered gradually, even though the protocol does not use the same Transfer-Encoding: chunked mechanism.

So the correct model is:

streaming HTTP response

and not:

chunked encoding = streaming

Backpressure

Streaming creates a new problem.

What if the backend produces data faster than the client consumes it?

producer >>> consumer

The buffer cannot grow forever.

Streams have a notion of backpressure: a slower consumer should affect the rate at which data is read and propagated through the pipeline.

But at the level of the whole system there is still a decision to make:

• buffer,
• slow the producer down,
• drop data,
• close the connection.

This is not an implementation detail. For realtime it can be part of the product semantics.

Where is raw HTTP streaming used?

Good cases include:

• token streaming from a language model,
• log output,
• progress of a long process,
• large results produced incrementally,
• NDJSON feeds.

If the communication is simply:

one request
-> incremental response
-> done

we do not need a WebSocket.

What are we still missing?

We have a long-lived response and we can send further records.

But we have to decide for ourselves:

How is an event framed?
Does it have an ID?
How does reconnect work?
How does the client resume?

For a one-way stream from the server to the browser, the web platform has a ready answer.

Server-Sent Events does not create a new transport.

It standardizes an event stream over HTTP.