Definition
Streaming generation is the practice of returning a model's output incrementally, one token (or a few tokens) at a time, while decoding is still running, instead of returning the full completion at the end. For LLM APIs the usual transport is server-sent events (SSE): a long-lived HTTP response with media type text/event-stream, defined in the WHATWG HTML Living Standard (originally the W3C EventSource specification).
Autoregressive decoding produces one token per forward pass (see Language modeling and KV cache). A 400-token answer at 50 tokens per second takes 8 seconds to finish. Without streaming the user stares at a spinner for all 8 seconds. With streaming the first words appear after about half a second and keep arriving faster than most people read.
Streaming sits between the inference server and the user interface. The server (vLLM, TGI or a hosted API) emits a frame after every decode step. Gateways and proxies must pass the frames through unbuffered. The client appends each delta to the screen, or feeds it to a parser, a text-to-speech engine or the next stage of an agent.
The latency arithmetic
- prompt length in tokens; prefill processes all of them in one parallel pass
- prefill throughput, typically thousands of tokens per second per request
- inter-token latency: gap between tokens and as seen by the client
- time per output token, the mean decode step; is the per-user token rate
- number of output tokens
Without streaming, the first visible token is the last one, so the perceived wait equals E2E. Streaming makes the perceived wait TTFT. It does not shorten E2E, and it does not change the token bill. Long prompts move TTFT through the prefill term, so streaming helps most when outputs are long and prompts are short; Prompt caching attacks the prefill term directly.
Wire formats you will meet
- SSE frame
- UTF-8 text lines
field: value; fields aredata,event,id,retry; a blank line ends one event. Lines starting with:are comments, often used as keep-alives. - Browser EventSource
- GET only, no request body, auto-reconnects with
Last-Event-ID. LLM clients usually POST withfetchand readresponse.bodyas a stream instead. - OpenAI-style chat
"stream": truereturnschat.completion.chunkobjects whosechoices[i].deltaholds newcontentortool_callsargument fragments; the stream ends withdata: [DONE]."stream_options": {"include_usage": true}adds a final chunk with token usage. vLLM's OpenAI-compatible server follows the same format.- Anthropic Messages
- Named events
message_start,content_block_start,content_block_delta(text_delta, orinput_json_deltafor tool arguments),content_block_stop,message_delta(stop reason and usage),message_stop, pluspingand in-banderrorevents. - TGI
- Native
/generate_streamendpoint emits one SSE event per token, alongside an OpenAI-compatible route.
What happens between the GPU and the screen
A frame is only useful if every hop forwards it at once. Reverse proxies that buffer responses (nginx does by default), response compression and some serverless platforms collect bytes until a buffer fills or the response ends. The tokens are then generated on time but delivered in bursts, or all at the end. For nginx the fixes are proxy_buffering off; on the route or an X-Accel-Buffering: no response header; compression should be off for text/event-stream.
The server also has to hold some text back. Detokenising one token at a time can split a multi-byte UTF-8 character, so servers withhold bytes until they form a complete character. With stop sequences, trailing text that could be the start of a stop string is withheld until it is confirmed not to be one; otherwise the client would see the first half of the stop string.
Throughput and streaming are separate. A server running Continuous batching decodes many requests in one step, so per-user TPOT grows with batch size even though total tokens per second rises. A prefill from a newly admitted request can delay everyone's next step; the viz models that as a periodic 60 ms stall. Chunked prefill exists to cap such stalls.
A robust streaming client
- 1POST with
stream: trueand read the body incrementally; split on blank lines and parse eachdata:payload. - 2Append text deltas to the display; accumulate tool-call and JSON fragments and parse them only when their block closes (or with a tolerant partial parser).
- 3Stop on the terminal frame (
[DONE],message_stop); treat a stream that ends without it, or an in-band error event, as a failure. - 4Record TTFT and inter-token gaps per request, and the final usage chunk for billing.
- 5On user cancel or page close, abort the request so the server frees the slot.
Worked example: what the user waits for
| Delivery | First text visible | Computation | Last token |
|---|---|---|---|
| Direct SSE | 0.52 s | 0.10 + 2000/5000 + 1/50 | 0.52 + 399 × 0.02 = 8.50 s |
| Proxy flushing every 64 bytes | 0.82 s | 16 tokens per flush: 0.52 + 15 × 0.02 | 8.50 s |
| Proxy with a 4 KB buffer | 8.50 s | 4096 / 4 = 1,024 tokens > 400, never fills | 8.50 s |
| No streaming | 8.50 s | whole body at the end | 8.50 s |
All four rows cost the same tokens and finish at the same moment. Only the first row turns an 8.5-second wait into a 0.5-second one; the 4 KB buffer silently undoes streaming completely.
1import time2from openai import OpenAI34client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")5t0 = time.perf_counter()6stream = client.chat.completions.create(7 model="my-model",8 messages=[{"role": "user", "content": "Explain SSE in three sentences."}],9 stream=True,10 stream_options={"include_usage": True}, # final chunk carries usage11)12arrivals, text = [], []13for chunk in stream:14 if chunk.choices and chunk.choices[0].delta.content:15 arrivals.append(time.perf_counter() - t0)16 text.append(chunk.choices[0].delta.content)17 print(chunk.choices[0].delta.content, end="", flush=True)18 if chunk.usage:19 n_out = chunk.usage.completion_tokens2021gaps = [b - a for a, b in zip(arrivals, arrivals[1:])]22print(f"\nTTFT {arrivals[0]:.3f} s, E2E {arrivals[-1]:.3f} s, "23 f"mean ITL {1000 * sum(gaps) / len(gaps):.1f} ms, max {1000 * max(gaps):.0f} ms, "24 f"{n_out} tokens")Use when
- +Perceived latency falls from E2E to TTFT, often by an order of magnitude for long answers.
- +Downstream stages (TTS, UI rendering, parsers) can start early.
- +Users can stop a bad answer early, and a client disconnect lets the server free the slot.
Avoid when
- −Every hop must be configured not to buffer; one misconfigured proxy silently undoes it.
- −Error handling, retries and moderation are harder once partial output has been shown.
- −A long-lived connection per request ties up gateway capacity and needs idle timeouts above the longest generation.
Check yourself
A chat app switches from non-streaming to streaming responses. Which quantity changes for a 400-token answer?
Check yourself
Server logs show TTFT of 0.4 s, but users behind the company gateway see the whole answer appear at once after 8 s. Most likely cause?
Remember
- ✓Streaming changes what the user waits for: TTFT instead of TTFT + (N−1)·TPOT. Tokens, cost and E2E stay the same.
- ✓TTFT = network + queue + prefill () + one step; long prompts raise it even when streaming.
- ✓Transport is usually SSE:
text/event-stream,data:lines, blank-line separators; OpenAI-style streams end withdata: [DONE], Anthropic's withmessage_stop. - ✓Buffering proxies and compression can turn a stream back into one burst; check every hop.
- ✓Handle mid-stream errors in-band, abort generation on client disconnect, and parse partial JSON or tool arguments incrementally.