Inference servers (vLLM, TGI)

An LLM inference server such as vLLM or TGI tokenizes requests, batches them continuously, packs a paged KV cache into spare GPU memory, runs the model and streams tokens, trading throughput against time to first token and per-token latency.

8 min read797 wordscore0 viewsupdated 2026-10-21

Definition

Inference server (serving engine) is the long-running process that holds a model's weights on accelerators and turns API requests into generated tokens. It owns tokenization, scheduling and batching, KV-cache memory, the forward pass, sampling and streaming. vLLM came from UC Berkeley (open-sourced June 2023; Kwon et al., SOSP 2023). Text Generation Inference (TGI) is Hugging Face's server, released in 2023. Other widely used engines are NVIDIA TensorRT-LLM, SGLang (Zheng et al., 2024) and, for local use, llama.cpp's server and Ollama.

Calling model.generate() inside a web handler works for a demo. Under real traffic it serves one request or one fixed batch at a time, reserves cache memory for the maximum length, and leaves most of the GPU idle while it waits for the longest answer. A serving engine adds the systems layer: Continuous batching to keep the batch full, PagedAttention to pack the KV cache, fused kernels, quantized weights, multi-GPU sharding and token streaming. In a product it sits behind a gateway that handles auth, rate limits and routing, and in front of the GPUs.

Request path

  1. 1API server. Accepts an OpenAI-style /v1/chat/completions (or a native) request, validates it, applies the model's chat template and tokenizes.
  2. 2Scheduler. Before every forward pass, decides which sequences run: running ones continue, finished ones leave, waiting ones are admitted while slots and KV blocks remain.
  3. 3KV block manager. Hands out cache blocks, reuses cached prefixes, and preempts sequences when memory runs out.
  4. 4Model executor. Runs one forward pass over the whole batch on one or more GPUs (tensor-parallel shards), using attention kernels that read the paged cache.
  5. 5Sampler and detokenizer. Applies temperature, top-p and the other sampling parameters, checks stop conditions and streams text deltas back as server-sent events (see Streaming generation).
clientPOST /v1/chat/completions{"model": "llama-2-13b-chat", "messages": [{"role": "user", "content": "Explain KV cache"}], "max_tokens": 256, "temperature": 0.7, "stream": true}streamed back (server-sent events)data: {"choices": [{"delta": {"content": " The"}}]}data: {"choices": [{"delta": {"content": " KV"}}]}…data: [DONE]inference server (vLLM, TGI, TensorRT-LLM, SGLang …)API serverOpenAI-style HTTP, validation, SSE streamingchat template + tokenizer → prompt token idsscheduler (every step)waiting R7 R8 R9running R1–R6 (≤ max_num_seqs)token budget per stepKV block manager3,750 blocks × 16prefix-cache hitspreempt when fullmodel executor (GPU workers)one forward pass for the whole running batchtensor-parallel shards · CUDA graphspaged attention kernels · fp16/bf16, fp8 or int4 weightssampler + detokenizerlogits → temperature / top-p → next token idstop strings, eos, max_tokens → text deltanew token → next stepvLLM: vllm serve <model> → Python engine + OpenAI-compatible server on port 8000TGI: launcher + Rust router (HTTP, batching) + Python model shards over gRPC
fig.A streaming chat request through a serving engine. The API server tokenizes; the scheduler and KV block manager choose the running batch each step (3,750 blocks of 16 tokens is the pool of Llama 2 13B at 0.9 memory utilisation on an 80 GiB GPU); the executor runs one pass for all sequences; the sampler emits one token per sequence, which is streamed to the client as an SSE chunk and fed back for the next step. vLLM runs this as a Python engine; TGI splits it into a Rust router and Python model shards.

What to measure

Te2e=TTFT+(nout−1)⋅TPOT,L=λ Te2eT_\text{e2e}=\text{TTFT}+(n_\text{out}-1)\cdot\text{TPOT},\qquad L=\lambda\,T_\text{e2e}
TTFT\text{TTFT}
time to first token: queueing + prefill of the prompt
TPOT\text{TPOT}
time per output token after the first (inter-token latency), set by the decode step time
L, λL,\ \lambda
Little's law: sequences in the system = arrival rate × time each spends there

Throughput (output tokens per second across all users) is the cost metric. TTFT and TPOT are the user-facing ones. They pull against each other: a larger batch raises throughput, but every sequence's step gets slower. Goodput counts only requests that meet both latency targets, and it is the number to optimise against.

NKV=⌊u⋅MGPU−Mweights−Mact2 Llayers Hkv dhead bytes⌋,tstep≳Mweights+∑ini⋅kvHBM bandwidthN_\text{KV}=\left\lfloor\frac{u\cdot M_\text{GPU}-M_\text{weights}-M_\text{act}}{2\,L_\text{layers}\,H_\text{kv}\,d_\text{head}\,\text{bytes}}\right\rfloor,\qquad t_\text{step}\gtrsim\frac{M_\text{weights}+\sum_i n_i\cdot\text{kv}}{\text{HBM bandwidth}}
uu
fraction of GPU memory the engine may use (vLLM --gpu-memory-utilization, default 0.9)
MactM_\text{act}
activation and workspace memory, measured at start-up by a profiling forward pass
NKVN_\text{KV}
tokens the KV pool can hold across all running sequences
tstept_\text{step}
decode step time: weights plus every live sequence's cache (nin_i tokens, kv bytes each) read once per step
AspectvLLMTGI
OriginUC Berkeley, 2023 (PagedAttention paper, SOSP 2023)Hugging Face, 2023
StructurePython engine on PyTorch with custom CUDA kernelsRust launcher and router (HTTP, batching) + Python model shards over gRPC
Startvllm serve <model> (port 8000)Docker image with --model-id <model> (port 80 in the container)
APIOpenAI-compatible /v1/completions, /v1/chat/completionsnative /generate, /generate_stream, and an OpenAI-compatible /v1/chat/completions
Batching and memorycontinuous batching, PagedAttention, prefix caching, chunked prefillcontinuous batching, FlashAttention and paged attention kernels
Multi-GPU--tensor-parallel-size, --pipeline-parallel-size--num-shard (tensor parallel)
QuantizationGPTQ, AWQ, FP8 and others; --kv-cache-dtype fp8--quantize with bitsandbytes, GPTQ, AWQ, EETQ, FP8 and others
LicenceApache 2.0Apache 2.0 (HFOIL 1.0 for v1.0, July 2023; Apache again from v2.0, April 2024)
vLLM and TGI in brief (both move quickly; check current docs before relying on a flag)
Knob (vLLM / TGI)RaisesCosts
--gpu-memory-utilization / --cuda-memory-fractionKV pool, so batch size and throughputless headroom; out-of-memory risk with other processes on the GPU
--max-num-seqs / --max-batch-total-tokensconcurrency and throughputTPOT grows with every added sequence
--max-num-batched-tokens / --max-batch-prefill-tokensprefill throughput per steplonger steps stall the decoding sequences
--max-model-len / --max-total-tokenslongest request servedfewer long requests fit at once
quantized weights or fp8 KV cachefree memory and bandwidth, so batch and speedsome accuracy; evaluate on your task (see Quantization)
The knobs that matter most, and what they trade

Where it's used

vLLM

general GPU serving with wide model coverage and an OpenAI-compatible API; also used as the generation engine inside RL and evaluation pipelines

TGI

Hugging Face deployments (Inference Endpoints, the Hub's model formats) with a Rust front end and Prometheus metrics

TensorRT-LLM

NVIDIA GPUs, with engines compiled ahead of time per model and GPU, usually behind Triton Inference Server

SGLang

workloads with heavy prefix reuse and structured, multi-call programs (RadixAttention keeps a tree of cached prefixes)

llama.cpp server, Ollama

quantized GGUF models on CPUs, Apple silicon and small GPUs, mostly single-user (see On-device generative AI)

Llama 2 13B (bf16) on one 80 GiB GPU, 2.0 TB/s · 200 requests per runweights 24.2 GiB · reserve 2 GiB · KV pool 45.8 GiB = 60,012 tokens · held back 8 GiB800 KiB of KV per token → 60 sequences of 1,000 tokens fit; max_num_seqs = 256output tokens/s vs offered load050010001481216TTFT mean (solid) and p90 (dashed), s07.5151481216request rate (req/s)request rate (req/s)at 6 req/soutput throughput848 tok/smean running batch31.4TTFT mean / p900.18 / 0.29 stime per output token38.6 mspeak KV pool use61%preemptions (recompute)0step = (weights + live KV bytes) / bandwidth + prefill FLOPs / 156 TFLOPS; FCFS admission;when the pool is full the newest sequence is preempted and recomputed later
model
max_num_seqs
fig.A simulated server on one 80 GiB GPU (2.0 TB/s, prefill at 156 TFLOPS) with 200 requests: prompts of 100–899 tokens, mean output 208 tokens. Defaults (Llama 2 13B, utilisation 0.9, max_num_seqs 256): the pool holds 60,012 tokens; at 6 req/s the server gives 848 tok/s, TTFT 0.18 s (p90 0.29 s), TPOT 38.6 ms and peak KV use 61%. At 10 req/s throughput saturates at 925 tok/s, the pool fills, 25 sequences are preempted, TPOT is 73.8 ms and p90 TTFT 5.46 s. max_num_seqs 32 at 10 req/s keeps TPOT at 32.9 ms but gives 803 tok/s and p90 TTFT 15.49 s. Llama 3 8B (GQA, 128 KiB per token) reaches 1,890 tok/s at 16 req/s using 15% of its pool.

Two walls appear in the figure. Up to about 8 req/s the server keeps up: throughput grows with load and TTFT stays near the prefill time. Past that, the GPU is saturated. Extra requests only lengthen the queue, so TTFT explodes while throughput flattens. For the 13B model the wall is memory: its 800 KiB per token fills the pool and forces preemption. Lowering utilisation to 0.6 cuts the pool to 28,554 tokens and raises preemptions at 10 req/s from 25 to 58.

Worked example: sizing one GPU

  1. 1Pool. Llama 2 13B in bf16 is 2×13.0B=26.02\times13.0\text{B}=26.0 GB = 24.21 GiB. With u=0.9u=0.9 of 80 GiB and 2 GiB of activations: 72−24.21−2=45.7972-24.21-2=45.79 GiB. At 800 KiB per token that is 60,012 tokens, or 3,750 blocks of 16.
  2. 2Target load. 5 req/s, mean output 208 tokens, TTFT 0.2 s and TPOT 40 ms. Then Te2e=0.2+207×0.04=8.48T_\text{e2e}=0.2+207\times0.04=8.48 s, and Little's law gives L=5×8.48=42.4L=5\times8.48=42.4 sequences in flight.
  3. 3Memory. Mean context per sequence is the prompt plus half the output: 484+104=588484+104=588 tokens (459 MiB). In total 42.4×588=24,93142.4\times588=24{,}931 tokens, 41.5% of the pool, so it fits.
  4. 4Speed. Pure decode reads 26.0+24,931×819,200/109=46.426.0+24{,}931\times819{,}200/10^9=46.4 GB per step, 23.2 ms at 2 TB/s. That is inside the 40 ms TPOT budget, which leaves time for prefills. The demand of 5×208=1,0405\times208=1{,}040 tok/s is below the ideal decode rate of 42.4/0.0232=1,82742.4/0.0232=1{,}827 tok/s.
python
1# vLLM: vllm serve meta-llama/Llama-2-13b-chat-hf --gpu-memory-utilization 0.9 --max-num-seqs 256
2# TGI: docker run --gpus all -p 8080:80 ghcr.io/huggingface/text-generation-inference:<version> \
3# --model-id meta-llama/Llama-2-13b-chat-hf (then use base_url http://localhost:8080/v1)
4import time
5from openai import OpenAI
6
7client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
8t0, first, chunks = time.perf_counter(), None, 0
9stream = client.chat.completions.create(
10 model="meta-llama/Llama-2-13b-chat-hf",
11 messages=[{"role": "user", "content": "Explain the KV cache in two sentences."}],
12 max_tokens=256, temperature=0.7, stream=True,
13)
14for chunk in stream:
15 if chunk.choices and chunk.choices[0].delta.content:
16 if first is None:
17 first = time.perf_counter() - t0 # TTFT
18 chunks += 1 # usually one token per chunk
19total = time.perf_counter() - t0
20print(f"TTFT {first:.3f} s, TPOT {1000 * (total - first) / max(chunks - 1, 1):.1f} ms, {chunks} chunks")
Launching vLLM or TGI and measuring TTFT and TPOT from a streaming OpenAI-compatible request (needs a GPU server and the openai package; not run here)

Use when

  • +Self-hosting an engine gives control of the model, data locality, and cost per token at high, steady utilisation.
  • +One OpenAI-compatible endpoint lets applications switch between hosted APIs and your own models.
  • +Engine features (prefix caching, quantization, speculative decoding) are configuration, not application code.

Avoid when

  • −GPUs cost the same idle or busy; at low or bursty traffic a hosted API is often cheaper.
  • −Latency targets, memory headroom and preemption need tuning and monitoring per model and GPU.
  • −Engines change quickly: flags, defaults and supported models move between releases.

Check yourself

A server's throughput is flat as the request rate rises, but TTFT keeps growing. What is happening?

Check yourself

Why does Llama 3 8B fit far more concurrent sequences than Llama 2 13B on the same GPU?

Remember

  • ✓A serving engine does API, tokenization, per-step scheduling, paged KV memory, batched forward passes, sampling and streaming.
  • ✓Latency =TTFT+(n−1) TPOT=\text{TTFT}+(n-1)\,\text{TPOT}; concurrency L=λTL=\lambda T (Little's law) sizes the KV memory you need.
  • ✓KV pool = utilisation × GPU memory − weights − activations; it caps the batch and hence throughput.
  • ✓Throughput rises with load until compute, bandwidth or KV memory saturates; then only the queue (TTFT) grows.
  • ✓vLLM: Python engine, PagedAttention, OpenAI API on port 8000. TGI: Rust router + Python shards, native and OpenAI APIs.
← back to Generative AI roadmap