Streaming generation

Streaming sends each decoded token to the client as it is produced, usually as server-sent events, so the wait drops from TTFT + (N−1)·TPOT to TTFT; total latency and cost are unchanged.

7 min read810 wordscore0 viewsupdated 2026-10-21

Definition

Streaming generation is the practice of returning a model's output incrementally, one token (or a few tokens) at a time, while decoding is still running, instead of returning the full completion at the end. For LLM APIs the usual transport is server-sent events (SSE): a long-lived HTTP response with media type text/event-stream, defined in the WHATWG HTML Living Standard (originally the W3C EventSource specification).

Autoregressive decoding produces one token per forward pass (see Language modeling and KV cache). A 400-token answer at 50 tokens per second takes 8 seconds to finish. Without streaming the user stares at a spinner for all 8 seconds. With streaming the first words appear after about half a second and keep arriving faster than most people read.

Streaming sits between the inference server and the user interface. The server (vLLM, TGI or a hosted API) emits a frame after every decode step. Gateways and proxies must pass the frames through unbuffered. The client appends each delta to the screen, or feeds it to a parser, a text-to-speech engine or the next stage of an agent.

The latency arithmetic

TTFT=tnet+tqueue+PRprefill+tstep,E2E=TTFT+∑i=2NITLi≈TTFT+(N−1) TPOT\text{TTFT}=t_\text{net}+t_\text{queue}+\frac{P}{R_\text{prefill}}+t_\text{step},\qquad \text{E2E}=\text{TTFT}+\sum_{i=2}^{N}\text{ITL}_i\approx\text{TTFT}+(N-1)\,\text{TPOT}
PP
prompt length in tokens; prefill processes all of them in one parallel pass
RprefillR_\text{prefill}
prefill throughput, typically thousands of tokens per second per request
ITLi\text{ITL}_i
inter-token latency: gap between tokens i−1i-1 and ii as seen by the client
TPOT\text{TPOT}
time per output token, the mean decode step; 1/TPOT1/\text{TPOT} is the per-user token rate
NN
number of output tokens

Without streaming, the first visible token is the last one, so the perceived wait equals E2E. Streaming makes the perceived wait TTFT. It does not shorten E2E, and it does not change the token bill. Long prompts move TTFT through the prefill term, so streaming helps most when outputs are long and prompts are short; Prompt caching attacks the prefill term directly.

one streamed request over HTTP server-sent events (SSE)clientserverPOST /chat/completions{"model": …, "messages": […], "stream": true}queue + network 0.10 sprefill: 2,000 prompt tokens in oneforward pass at 5,000 tok/s 0.40 s(fills the KV cache, emits token 1)HTTP 200 · Content-Type: text/event-streamdata: {"choices":[{"delta":{"role":"assistant"}}]}data: {"choices":[{"delta":{"content":"Stream"}}]}data: {"choices":[{"delta":{"content":"ing"}}]}data: {"choices":[{"delta":{"content":" lets"}}]}data: {"choices":[{"delta":{},"finish_reason":"stop"}]}data: [DONE]… one frame per decode step, blank line after eachTTFT0.52 sITL 20 msE2Esame idea, Anthropic Messages API: named events (event: + data: lines)1. message_start2. content_block_start3. content_block_delta × n4. content_block_stop5. message_delta6. message_stopE2E = TTFT + (N − 1) × ITL; streaming moves the first token earlier, not the last
fig.One streamed request. The client POSTs with "stream": true; the server spends 0.10 s on network and queueing and 0.40 s prefilling 2,000 prompt tokens at 5,000 tok/s, then answers HTTP 200 with Content-Type text/event-stream. Each decode step becomes a data: frame carrying a delta; the stream ends with a finish_reason frame and data: [DONE]. TTFT is 0.52 s; frames then arrive every ~20 ms. Below, the Anthropic Messages API's named events for the same flow.

Wire formats you will meet

SSE frame
UTF-8 text lines field: value; fields are data, event, id, retry; a blank line ends one event. Lines starting with : are comments, often used as keep-alives.
Browser EventSource
GET only, no request body, auto-reconnects with Last-Event-ID. LLM clients usually POST with fetch and read response.body as a stream instead.
OpenAI-style chat
"stream": true returns chat.completion.chunk objects whose choices[i].delta holds new content or tool_calls argument fragments; the stream ends with data: [DONE]. "stream_options": {"include_usage": true} adds a final chunk with token usage. vLLM's OpenAI-compatible server follows the same format.
Anthropic Messages
Named events message_start, content_block_start, content_block_delta (text_delta, or input_json_delta for tool arguments), content_block_stop, message_delta (stop reason and usage), message_stop, plus ping and in-band error events.
TGI
Native /generate_stream endpoint emits one SSE event per token, alongside an OpenAI-compatible route.

What happens between the GPU and the screen

A frame is only useful if every hop forwards it at once. Reverse proxies that buffer responses (nginx does by default), response compression and some serverless platforms collect bytes until a buffer fills or the response ends. The tokens are then generated on time but delivered in bursts, or all at the end. For nginx the fixes are proxy_buffering off; on the route or an X-Accel-Buffering: no response header; compression should be off for text/event-stream.

The server also has to hold some text back. Detokenising one token at a time can split a multi-byte UTF-8 character, so servers withhold bytes until they form a complete character. With stop sequences, trailing text that could be the start of a stop string is withheld until it is confirmed not to be one; otherwise the client would see the first half of the stop string.

40 output tokens · prompt 2000 tokens · decode 50 tok/s · showing direct SSEserver decodeclient: SSEclient: proxyclient: no streamprefill 0.40 s0.00.51.01.52.0time (s)screen at t = 0.90 s: 16/40 tokens visible, 16/40 generated{"title": "Streaming generation", "tags": ["sse", "ttft",incremental JSON parse (close open string/brackets, then JSON.parse)done title: "Streaming generation"ready 0.62 spartialtags: ["sse","ttft"]ready 0.98 sabsent summary: —ready 1.54 sTTFT0.52 sE2E1.54 slongest gap0.08 sserver TPOT26.2 msTTFT = 0.10 s network/queue + prefill + one decode step; every 8th step stalls 60 ms
delivery
fig.A 40-token JSON completion. At the defaults (2,000-token prompt, 50 tok/s, direct SSE) TTFT is 0.52 s and E2E 1.54 s; the parsed title is complete at 0.62 s and tags at 0.98 s, and at t = 0.90 s 16 of 40 tokens are on screen. Behind a proxy that flushes 64-byte chunks, TTFT becomes 1.00 s and the longest gap 0.44 s; without streaming everything appears at 1.54 s. A 16,000-token prompt adds 2.80 s of prefill (direct-SSE TTFT 3.32 s); at 20 tok/s the server TPOT rises from 26.2 ms to 56.2 ms (both include a 60 ms stall every 8th step).

Throughput and streaming are separate. A server running Continuous batching decodes many requests in one step, so per-user TPOT grows with batch size even though total tokens per second rises. A prefill from a newly admitted request can delay everyone's next step; the viz models that as a periodic 60 ms stall. Chunked prefill exists to cap such stalls.

A robust streaming client

  1. 1POST with stream: true and read the body incrementally; split on blank lines and parse each data: payload.
  2. 2Append text deltas to the display; accumulate tool-call and JSON fragments and parse them only when their block closes (or with a tolerant partial parser).
  3. 3Stop on the terminal frame ([DONE], message_stop); treat a stream that ends without it, or an in-band error event, as a failure.
  4. 4Record TTFT and inter-token gaps per request, and the final usage chunk for billing.
  5. 5On user cancel or page close, abort the request so the server frees the slot.

Worked example: what the user waits for

DeliveryFirst text visibleComputationLast token
Direct SSE0.52 s0.10 + 2000/5000 + 1/500.52 + 399 × 0.02 = 8.50 s
Proxy flushing every 64 bytes0.82 s16 tokens per flush: 0.52 + 15 × 0.028.50 s
Proxy with a 4 KB buffer8.50 s4096 / 4 = 1,024 tokens > 400, never fills8.50 s
No streaming8.50 swhole body at the end8.50 s
2,000-token prompt, prefill 5,000 tok/s, 0.10 s network + queue, decode 50 tok/s (20 ms per token), 400-token answer, about 4 bytes per token

All four rows cost the same tokens and finish at the same moment. Only the first row turns an 8.5-second wait into a 0.5-second one; the 4 KB buffer silently undoes streaming completely.

python
1import time
2from openai import OpenAI
3
4client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
5t0 = time.perf_counter()
6stream = client.chat.completions.create(
7 model="my-model",
8 messages=[{"role": "user", "content": "Explain SSE in three sentences."}],
9 stream=True,
10 stream_options={"include_usage": True}, # final chunk carries usage
11)
12arrivals, text = [], []
13for chunk in stream:
14 if chunk.choices and chunk.choices[0].delta.content:
15 arrivals.append(time.perf_counter() - t0)
16 text.append(chunk.choices[0].delta.content)
17 print(chunk.choices[0].delta.content, end="", flush=True)
18 if chunk.usage:
19 n_out = chunk.usage.completion_tokens
20
21gaps = [b - a for a, b in zip(arrivals, arrivals[1:])]
22print(f"\nTTFT {arrivals[0]:.3f} s, E2E {arrivals[-1]:.3f} s, "
23 f"mean ITL {1000 * sum(gaps) / len(gaps):.1f} ms, max {1000 * max(gaps):.0f} ms, "
24 f"{n_out} tokens")
Measure TTFT and inter-token gaps against any OpenAI-compatible server (e.g. vLLM on localhost). Uses the openai SDK, which was not run here

Use when

  • +Perceived latency falls from E2E to TTFT, often by an order of magnitude for long answers.
  • +Downstream stages (TTS, UI rendering, parsers) can start early.
  • +Users can stop a bad answer early, and a client disconnect lets the server free the slot.

Avoid when

  • −Every hop must be configured not to buffer; one misconfigured proxy silently undoes it.
  • −Error handling, retries and moderation are harder once partial output has been shown.
  • −A long-lived connection per request ties up gateway capacity and needs idle timeouts above the longest generation.

Check yourself

A chat app switches from non-streaming to streaming responses. Which quantity changes for a 400-token answer?

Check yourself

Server logs show TTFT of 0.4 s, but users behind the company gateway see the whole answer appear at once after 8 s. Most likely cause?

Remember

  • ✓Streaming changes what the user waits for: TTFT instead of TTFT + (N−1)·TPOT. Tokens, cost and E2E stay the same.
  • ✓TTFT = network + queue + prefill (P/RprefillP/R_\text{prefill}) + one step; long prompts raise it even when streaming.
  • ✓Transport is usually SSE: text/event-stream, data: lines, blank-line separators; OpenAI-style streams end with data: [DONE], Anthropic's with message_stop.
  • ✓Buffering proxies and compression can turn a stream back into one burst; check every hop.
  • ✓Handle mid-stream errors in-band, abort generation on client disconnect, and parse partial JSON or tool arguments incrementally.
← back to Generative AI roadmap