# Streaming generation

> Streaming sends each decoded token to the client as it is produced, usually as server-sent events, so the wait drops from TTFT + (N−1)·TPOT to TTFT; total latency and cost are unchanged.

Source: https://www.mlfoundry.in/learn/streaming-generation  
Section: Generative AI › Serving Generative Models  
Level: core · 7 min read · updated 2026-10-21

**Streaming generation:** is the practice of returning a model's output incrementally, one token (or a few tokens) at a time, while decoding is still running, instead of returning the full completion at the end. For LLM APIs the usual transport is **server-sent events** (SSE): a long-lived HTTP response with media type `text/event-stream`, defined in the WHATWG HTML Living Standard (originally the W3C EventSource specification).

Autoregressive decoding produces one token per forward pass (see **Language modeling** and **KV cache**). A 400-token answer at 50 tokens per second takes 8 seconds to finish. Without streaming the user stares at a spinner for all 8 seconds. With streaming the first words appear after about half a second and keep arriving faster than most people read.

Streaming sits between the inference server and the user interface. The server (vLLM, TGI or a hosted API) emits a frame after every decode step. Gateways and proxies must pass the frames through unbuffered. The client appends each delta to the screen, or feeds it to a parser, a text-to-speech engine or the next stage of an agent.

## The latency arithmetic

$$
\text{TTFT}=t_\text{net}+t_\text{queue}+\frac{P}{R_\text{prefill}}+t_\text{step},\qquad \text{E2E}=\text{TTFT}+\sum_{i=2}^{N}\text{ITL}_i\approx\text{TTFT}+(N-1)\,\text{TPOT}
$$

- $P$: prompt length in tokens; prefill processes all of them in one parallel pass
- $R_\text{prefill}$: prefill throughput, typically thousands of tokens per second per request
- $\text{ITL}_i$: inter-token latency: gap between tokens $i-1$ and $i$ as seen by the client
- $\text{TPOT}$: time per output token, the mean decode step; $1/\text{TPOT}$ is the per-user token rate
- $N$: number of output tokens

Without streaming, the first visible token *is* the last one, so the perceived wait equals E2E. Streaming makes the perceived wait TTFT. It does not shorten E2E, and it does not change the token bill. Long prompts move TTFT through the prefill term, so streaming helps most when outputs are long and prompts are short; **Prompt caching** attacks the prefill term directly.

> Figure: One streamed request. The client POSTs with "stream": true; the server spends 0.10 s on network and queueing and 0.40 s prefilling 2,000 prompt tokens at 5,000 tok/s, then answers HTTP 200 with Content-Type text/event-stream. Each decode step becomes a data: frame carrying a delta; the stream ends with a finish_reason frame and data: [DONE]. TTFT is 0.52 s; frames then arrive every ~20 ms. Below, the Anthropic Messages API's named events for the same flow.

**Wire formats you will meet**

- **SSE frame:** UTF-8 text lines `field: value`; fields are `data`, `event`, `id`, `retry`; a blank line ends one event. Lines starting with `:` are comments, often used as keep-alives.
- **Browser EventSource:** GET only, no request body, auto-reconnects with `Last-Event-ID`. LLM clients usually POST with `fetch` and read `response.body` as a stream instead.
- **OpenAI-style chat:** `"stream": true` returns `chat.completion.chunk` objects whose `choices[i].delta` holds new `content` or `tool_calls` argument fragments; the stream ends with `data: [DONE]`. `"stream_options": {"include_usage": true}` adds a final chunk with token usage. vLLM's OpenAI-compatible server follows the same format.
- **Anthropic Messages:** Named events `message_start`, `content_block_start`, `content_block_delta` (`text_delta`, or `input_json_delta` for tool arguments), `content_block_stop`, `message_delta` (stop reason and usage), `message_stop`, plus `ping` and in-band `error` events.
- **TGI:** Native `/generate_stream` endpoint emits one SSE event per token, alongside an OpenAI-compatible route.

## What happens between the GPU and the screen

A frame is only useful if every hop forwards it at once. Reverse proxies that buffer responses (nginx does by default), response compression and some serverless platforms collect bytes until a buffer fills or the response ends. The tokens are then generated on time but delivered in bursts, or all at the end. For nginx the fixes are `proxy_buffering off;` on the route or an `X-Accel-Buffering: no` response header; compression should be off for `text/event-stream`.

The server also has to hold some text back. Detokenising one token at a time can split a multi-byte UTF-8 character, so servers withhold bytes until they form a complete character. With stop sequences, trailing text that could be the start of a stop string is withheld until it is confirmed not to be one; otherwise the client would see the first half of the stop string.

> Figure: A 40-token JSON completion. At the defaults (2,000-token prompt, 50 tok/s, direct SSE) TTFT is 0.52 s and E2E 1.54 s; the parsed title is complete at 0.62 s and tags at 0.98 s, and at t = 0.90 s 16 of 40 tokens are on screen. Behind a proxy that flushes 64-byte chunks, TTFT becomes 1.00 s and the longest gap 0.44 s; without streaming everything appears at 1.54 s. A 16,000-token prompt adds 2.80 s of prefill (direct-SSE TTFT 3.32 s); at 20 tok/s the server TPOT rises from 26.2 ms to 56.2 ms (both include a 60 ms stall every 8th step).

Throughput and streaming are separate. A server running **Continuous batching** decodes many requests in one step, so per-user TPOT grows with batch size even though total tokens per second rises. A prefill from a newly admitted request can delay everyone's next step; the viz models that as a periodic 60 ms stall. Chunked prefill exists to cap such stalls.

**A robust streaming client**

1. POST with `stream: true` and read the body incrementally; split on blank lines and parse each `data:` payload.
2. Append text deltas to the display; accumulate tool-call and JSON fragments and parse them only when their block closes (or with a tolerant partial parser).
3. Stop on the terminal frame (`[DONE]`, `message_stop`); treat a stream that ends without it, or an in-band error event, as a failure.
4. Record TTFT and inter-token gaps per request, and the final usage chunk for billing.
5. On user cancel or page close, abort the request so the server frees the slot.

## Worked example: what the user waits for

2,000-token prompt, prefill 5,000 tok/s, 0.10 s network + queue, decode 50 tok/s (20 ms per token), 400-token answer, about 4 bytes per token

| Delivery | First text visible | Computation | Last token |
| --- | --- | --- | --- |
| Direct SSE | 0.52 s | 0.10 + 2000/5000 + 1/50 | 0.52 + 399 × 0.02 = 8.50 s |
| Proxy flushing every 64 bytes | 0.82 s | 16 tokens per flush: 0.52 + 15 × 0.02 | 8.50 s |
| Proxy with a 4 KB buffer | 8.50 s | 4096 / 4 = 1,024 tokens > 400, never fills | 8.50 s |
| No streaming | 8.50 s | whole body at the end | 8.50 s |

All four rows cost the same tokens and finish at the same moment. Only the first row turns an 8.5-second wait into a 0.5-second one; the 4 KB buffer silently undoes streaming completely.

```python
import time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
t0 = time.perf_counter()
stream = client.chat.completions.create(
    model="my-model",
    messages=[{"role": "user", "content": "Explain SSE in three sentences."}],
    stream=True,
    stream_options={"include_usage": True},   # final chunk carries usage
)
arrivals, text = [], []
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        arrivals.append(time.perf_counter() - t0)
        text.append(chunk.choices[0].delta.content)
        print(chunk.choices[0].delta.content, end="", flush=True)
    if chunk.usage:
        n_out = chunk.usage.completion_tokens

gaps = [b - a for a, b in zip(arrivals, arrivals[1:])]
print(f"\nTTFT {arrivals[0]:.3f} s, E2E {arrivals[-1]:.3f} s, "
      f"mean ITL {1000 * sum(gaps) / len(gaps):.1f} ms, max {1000 * max(gaps):.0f} ms, "
      f"{n_out} tokens")
```

Measure TTFT and inter-token gaps against any OpenAI-compatible server (e.g. vLLM on localhost). Uses the openai SDK, which was not run here

> **Errors after the 200:** Once the server has sent `200 OK` and the first frames, it can no longer change the status code. A failure mid-stream (overload, a dropped connection, a safety stop) has to arrive as an in-band error event or a truncated stream with no final `finish_reason`. Clients must treat a stream that ends without its terminal frame as failed. When the client disconnects, the server should abort the request; otherwise it keeps decoding tokens nobody will read.

> **Structured output and tools stream too:** JSON output and tool-call arguments arrive as string fragments. Closing the open string and brackets and re-parsing gives a usable partial object, as the viz shows, so a UI can render the `title` field before the `summary` exists. Act on a tool call only after its block is complete. Output **Guardrails** face the same problem in reverse: text already shown cannot be retracted, so streamed moderation checks chunks as they go or delays display by a few sentences.

**Strengths**

- Perceived latency falls from E2E to TTFT, often by an order of magnitude for long answers.
- Downstream stages (TTS, UI rendering, parsers) can start early.
- Users can stop a bad answer early, and a client disconnect lets the server free the slot.

**Limitations**

- Every hop must be configured not to buffer; one misconfigured proxy silently undoes it.
- Error handling, retries and moderation are harder once partial output has been shown.
- A long-lived connection per request ties up gateway capacity and needs idle timeouts above the longest generation.

**Question:** A chat app switches from non-streaming to streaming responses. Which quantity changes for a 400-token answer?

- A. The number of billed output tokens
- B. End-to-end latency to the last token
- C. Time until the user sees the first words
- D. GPU time per request

**Answer:** C. Time until the user sees the first words — The server does the same prefill and the same 400 decode steps, so tokens, GPU time and E2E are unchanged. Only delivery changes: the first words arrive after TTFT (about 0.5 s in the worked example) instead of after E2E (8.5 s).

**Question:** Server logs show TTFT of 0.4 s, but users behind the company gateway see the whole answer appear at once after 8 s. Most likely cause?

- A. The model's decode rate dropped
- B. A proxy or compression layer is buffering the event stream
- C. The prompt is too long
- D. The client is using EventSource instead of fetch

**Answer:** B. A proxy or compression layer is buffering the event stream — Generation is on time at the server, so the delay is in delivery. A buffering reverse proxy or gzip on `text/event-stream` holds the frames until its buffer fills or the response ends. Disable buffering on that route (`proxy_buffering off` or `X-Accel-Buffering: no` for nginx).

**Key points**

- Streaming changes what the user waits for: TTFT instead of TTFT + (N−1)·TPOT. Tokens, cost and E2E stay the same.
- TTFT = network + queue + prefill ($P/R_\text{prefill}$) + one step; long prompts raise it even when streaming.
- Transport is usually SSE: `text/event-stream`, `data:` lines, blank-line separators; OpenAI-style streams end with `data: [DONE]`, Anthropic's with `message_stop`.
- Buffering proxies and compression can turn a stream back into one burst; check every hop.
- Handle mid-stream errors in-band, abort generation on client disconnect, and parse partial JSON or tool arguments incrementally.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
