Definition
Continuous batching (iteration-level scheduling; NVIDIA TensorRT-LLM calls it in-flight batching) is a way of serving autoregressive models in which the scheduler forms a new batch before every forward pass. A sequence leaves the batch as soon as it emits its end token, and a waiting request joins at the next iteration. It was introduced by Orca (Yu, Jeong, Kim, Kim and Chun, "Orca: A Distributed Serving System for Transformer-Based Generative Models", OSDI 2022). The name "continuous batching" spread with Hugging Face TGI and vLLM in 2023.
A language model generates one token per forward pass, and requests need very different numbers of passes: a yes/no answer needs 2, an essay 800. Classic model servers batch at the request level. They collect up to requests, run them together, and return when the batch is done. That suits image classifiers, where every input costs one pass. For generation it fails twice: short requests sit in finished slots until the longest one ends, and new arrivals wait for the whole batch to drain.
Batching itself is what makes decoding cheap. Each decode step reads every weight from GPU memory once, whether the batch holds 1 sequence or 64, and the matrix units are mostly idle at batch 1 (see KV cache). So the goal is to keep every slot busy with a real sequence at every step. Continuous batching is the scheduler that does this. Every serving engine covered in Inference servers (vLLM, TGI) uses it.
The waste in request-level batching
- output tokens of request i in the batch
- share of slot-steps that produce a real token; the rest is padding
- time of one decode iteration at batch size B; padded slots still cost compute
With outputs of 4, 9, 6 and 3 tokens, . Output lengths in chat traffic are long-tailed, so one long answer per batch is enough to idle most slots. Every request in the batch also gets its response at , the time of the longest one.
Scheduling every iteration
- 1Before each forward pass, drop sequences that emitted an end token or hit their length limit, and return their responses.
- 2Fill the free slots from the queue (first come, first served), as far as the slot limit and free KV cache memory allow.
- 3Run one pass over the whole batch. Newly admitted requests run their prefill (the full prompt) in this pass. Running ones decode one token each.
- 4Append each new token to its sequence (and stream it to the client), then go back to step 1.
Ragged batches are the hard part to implement. The sequences in one iteration have different lengths, and new ones bring whole prompts. Orca's selective batching stacks all tokens of the iteration into one matrix for the QKV projections and MLPs, which do not care which sequence a token belongs to. It splits them only for attention, which runs per sequence over that sequence's own keys and values. Modern engines go further with variable-length attention kernels over a paged cache (see PagedAttention), so no padding is ever materialised.
Prefill gets in the way
- fixed cost of reading the weights once per iteration (memory-bound)
- per-sequence cost: reading its KV cache and its share of compute
- prefill of the prompts admitted this iteration (compute-bound)
A 4,000-token prompt joining the batch makes that iteration many times slower than a pure decode step, so every running sequence sees a gap between two tokens. Chunked prefill caps the tokens per iteration (vLLM's max_num_batched_tokens) and spreads a long prompt over several iterations, mixed with decodes. Sarathi-Serve (Agrawal et al., OSDI 2024) and DeepSpeed-FastGen's Dynamic SplitFuse (2023) build on this. Disaggregated serving (DistServe, Splitwise, 2024) instead runs prefill and decode on separate GPUs.
Reported results and engine behaviour
- Orca (OSDI 2022)
- 36.9× the throughput of NVIDIA FasterTransformer at the same latency, on GPT-3 175B; FasterTransformer batched per request
- vLLM (SOSP 2023)
- 2–4× the throughput of FasterTransformer and Orca at the same latency; the gain comes from fitting more sequences per batch with paged KV memory, on top of iteration-level scheduling
- Hugging Face TGI
- the Rust router batches continuously;
--max-batch-total-tokenscaps the tokens (prompt + generated) held by one batch - vLLM
--max-num-seqscaps sequences per iteration;--max-num-batched-tokenscaps tokens per iteration, which bounds the prefill cost- TensorRT-LLM
- in-flight batching in the C++ batch manager, the same idea under another name
The figure shows the two regimes. Under light load the server is idle part of the time, so continuous batching mostly cuts latency: nobody waits for someone else's long answer. Under heavy load it raises throughput, because the slots stay full. At 4 slots and 8 req/s the continuous scheduler is 94.4% busy and the queue wait grows to 3.15 s. Then the only fix is more slots, which means more KV memory.
Worked example: the diagram, timed
| Measure | Static | Continuous |
|---|---|---|
| Iterations | 9 at b = 4, then 5 at b = 2 (14) | 6 at b = 4, 2 at b = 3, 1 at b = 1 (9) |
| Time | 9·13 + 5·12.5 = 179.5 ms | 6·13 + 2·12.75 + 12.25 = 115.75 ms |
| Throughput | 31 / 0.1795 s = 172.7 tok/s | 31 / 0.11575 s = 267.8 tok/s |
| Busy slot-steps | 31 / 56 = 55.4% | 31 / 36 = 86.1% |
| R5 first token | iteration 10 | iteration 4 |
1from collections import deque23# (name, arrival iteration, output tokens) as in the diagram4reqs = [("R1", 1, 4), ("R2", 1, 9), ("R3", 1, 6), ("R4", 1, 3), ("R5", 2, 5), ("R6", 3, 4)]5B = 4 # max sequences per iteration67def serve(reqs, B):8 pending, queue, running = deque(reqs), deque(), {}9 it = 010 while pending or queue or running:11 it += 112 while pending and pending[0][1] <= it: # arrivals13 queue.append(pending.popleft())14 while queue and len(running) < B: # refill free slots15 name, _, n_out = queue.popleft()16 running[name] = [0, n_out, it] # prefill runs in this pass17 for name in list(running): # one forward pass18 st = running[name]19 st[0] += 1 # one new token each20 if st[0] == st[1]: # eos: leave immediately21 print(f"{name}: iterations {st[2]}-{it}")22 del running[name]23 return it2425print("total iterations:", serve(reqs, B)) # 9 (static batching: 14)Use when
- +No padding steps: throughput rises with load until KV memory or compute saturates.
- +Short requests return as soon as they finish, so latency tracks each request's own length.
- +New requests start at the next iteration instead of after the current batch.
Avoid when
- −Needs per-sequence KV state, ragged-batch kernels and a scheduler inside the generation loop.
- −Prefills mixed into decode steps cause token-latency spikes unless prefill is chunked.
- −Admitting greedily can overcommit memory and force preemption.
Check yourself
A static batch holds four requests that generate 10, 10, 10 and 200 tokens. What share of its slot-steps produce real tokens?
Check yourself
Under continuous batching, why does a long new prompt slow down the other users' streams?
Remember
- ✓Schedule per iteration, not per request: finished sequences leave immediately, waiting ones join at the next forward pass.
- ✓Request-level batching wastes of its slot-steps and returns every response at the time of the longest one.
- ✓Selective batching: linear layers run on all tokens stacked; attention runs per sequence (now via variable-length, paged kernels).
- ✓Light load: mainly lower latency. Heavy load: higher throughput, until KV memory limits the slots.
- ✓Prefill in a decode batch stalls everyone; chunked prefill and prefill/decode disaggregation address it.