Definition
Inference server (serving engine) is the long-running process that holds a model's weights on accelerators and turns API requests into generated tokens. It owns tokenization, scheduling and batching, KV-cache memory, the forward pass, sampling and streaming. vLLM came from UC Berkeley (open-sourced June 2023; Kwon et al., SOSP 2023). Text Generation Inference (TGI) is Hugging Face's server, released in 2023. Other widely used engines are NVIDIA TensorRT-LLM, SGLang (Zheng et al., 2024) and, for local use, llama.cpp's server and Ollama.
Calling model.generate() inside a web handler works for a demo. Under real traffic it serves one request or one fixed batch at a time, reserves cache memory for the maximum length, and leaves most of the GPU idle while it waits for the longest answer. A serving engine adds the systems layer: Continuous batching to keep the batch full, PagedAttention to pack the KV cache, fused kernels, quantized weights, multi-GPU sharding and token streaming. In a product it sits behind a gateway that handles auth, rate limits and routing, and in front of the GPUs.
Request path
- 1API server. Accepts an OpenAI-style
/v1/chat/completions(or a native) request, validates it, applies the model's chat template and tokenizes. - 2Scheduler. Before every forward pass, decides which sequences run: running ones continue, finished ones leave, waiting ones are admitted while slots and KV blocks remain.
- 3KV block manager. Hands out cache blocks, reuses cached prefixes, and preempts sequences when memory runs out.
- 4Model executor. Runs one forward pass over the whole batch on one or more GPUs (tensor-parallel shards), using attention kernels that read the paged cache.
- 5Sampler and detokenizer. Applies temperature, top-p and the other sampling parameters, checks stop conditions and streams text deltas back as server-sent events (see Streaming generation).
What to measure
- time to first token: queueing + prefill of the prompt
- time per output token after the first (inter-token latency), set by the decode step time
- Little's law: sequences in the system = arrival rate × time each spends there
Throughput (output tokens per second across all users) is the cost metric. TTFT and TPOT are the user-facing ones. They pull against each other: a larger batch raises throughput, but every sequence's step gets slower. Goodput counts only requests that meet both latency targets, and it is the number to optimise against.
- fraction of GPU memory the engine may use (vLLM
--gpu-memory-utilization, default 0.9) - activation and workspace memory, measured at start-up by a profiling forward pass
- tokens the KV pool can hold across all running sequences
- decode step time: weights plus every live sequence's cache ( tokens, kv bytes each) read once per step
| Aspect | vLLM | TGI |
|---|---|---|
| Origin | UC Berkeley, 2023 (PagedAttention paper, SOSP 2023) | Hugging Face, 2023 |
| Structure | Python engine on PyTorch with custom CUDA kernels | Rust launcher and router (HTTP, batching) + Python model shards over gRPC |
| Start | vllm serve <model> (port 8000) | Docker image with --model-id <model> (port 80 in the container) |
| API | OpenAI-compatible /v1/completions, /v1/chat/completions | native /generate, /generate_stream, and an OpenAI-compatible /v1/chat/completions |
| Batching and memory | continuous batching, PagedAttention, prefix caching, chunked prefill | continuous batching, FlashAttention and paged attention kernels |
| Multi-GPU | --tensor-parallel-size, --pipeline-parallel-size | --num-shard (tensor parallel) |
| Quantization | GPTQ, AWQ, FP8 and others; --kv-cache-dtype fp8 | --quantize with bitsandbytes, GPTQ, AWQ, EETQ, FP8 and others |
| Licence | Apache 2.0 | Apache 2.0 (HFOIL 1.0 for v1.0, July 2023; Apache again from v2.0, April 2024) |
| Knob (vLLM / TGI) | Raises | Costs |
|---|---|---|
--gpu-memory-utilization / --cuda-memory-fraction | KV pool, so batch size and throughput | less headroom; out-of-memory risk with other processes on the GPU |
--max-num-seqs / --max-batch-total-tokens | concurrency and throughput | TPOT grows with every added sequence |
--max-num-batched-tokens / --max-batch-prefill-tokens | prefill throughput per step | longer steps stall the decoding sequences |
--max-model-len / --max-total-tokens | longest request served | fewer long requests fit at once |
| quantized weights or fp8 KV cache | free memory and bandwidth, so batch and speed | some accuracy; evaluate on your task (see Quantization) |
Where it's used
vLLM
general GPU serving with wide model coverage and an OpenAI-compatible API; also used as the generation engine inside RL and evaluation pipelines
TGI
Hugging Face deployments (Inference Endpoints, the Hub's model formats) with a Rust front end and Prometheus metrics
TensorRT-LLM
NVIDIA GPUs, with engines compiled ahead of time per model and GPU, usually behind Triton Inference Server
SGLang
workloads with heavy prefix reuse and structured, multi-call programs (RadixAttention keeps a tree of cached prefixes)
llama.cpp server, Ollama
quantized GGUF models on CPUs, Apple silicon and small GPUs, mostly single-user (see On-device generative AI)
Two walls appear in the figure. Up to about 8 req/s the server keeps up: throughput grows with load and TTFT stays near the prefill time. Past that, the GPU is saturated. Extra requests only lengthen the queue, so TTFT explodes while throughput flattens. For the 13B model the wall is memory: its 800 KiB per token fills the pool and forces preemption. Lowering utilisation to 0.6 cuts the pool to 28,554 tokens and raises preemptions at 10 req/s from 25 to 58.
Worked example: sizing one GPU
- 1Pool. Llama 2 13B in bf16 is GB = 24.21 GiB. With of 80 GiB and 2 GiB of activations: GiB. At 800 KiB per token that is 60,012 tokens, or 3,750 blocks of 16.
- 2Target load. 5 req/s, mean output 208 tokens, TTFT 0.2 s and TPOT 40 ms. Then s, and Little's law gives sequences in flight.
- 3Memory. Mean context per sequence is the prompt plus half the output: tokens (459 MiB). In total tokens, 41.5% of the pool, so it fits.
- 4Speed. Pure decode reads GB per step, 23.2 ms at 2 TB/s. That is inside the 40 ms TPOT budget, which leaves time for prefills. The demand of tok/s is below the ideal decode rate of tok/s.
1# vLLM: vllm serve meta-llama/Llama-2-13b-chat-hf --gpu-memory-utilization 0.9 --max-num-seqs 2562# TGI: docker run --gpus all -p 8080:80 ghcr.io/huggingface/text-generation-inference:<version> \3# --model-id meta-llama/Llama-2-13b-chat-hf (then use base_url http://localhost:8080/v1)4import time5from openai import OpenAI67client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")8t0, first, chunks = time.perf_counter(), None, 09stream = client.chat.completions.create(10 model="meta-llama/Llama-2-13b-chat-hf",11 messages=[{"role": "user", "content": "Explain the KV cache in two sentences."}],12 max_tokens=256, temperature=0.7, stream=True,13)14for chunk in stream:15 if chunk.choices and chunk.choices[0].delta.content:16 if first is None:17 first = time.perf_counter() - t0 # TTFT18 chunks += 1 # usually one token per chunk19total = time.perf_counter() - t020print(f"TTFT {first:.3f} s, TPOT {1000 * (total - first) / max(chunks - 1, 1):.1f} ms, {chunks} chunks")Use when
- +Self-hosting an engine gives control of the model, data locality, and cost per token at high, steady utilisation.
- +One OpenAI-compatible endpoint lets applications switch between hosted APIs and your own models.
- +Engine features (prefix caching, quantization, speculative decoding) are configuration, not application code.
Avoid when
- −GPUs cost the same idle or busy; at low or bursty traffic a hosted API is often cheaper.
- −Latency targets, memory headroom and preemption need tuning and monitoring per model and GPU.
- −Engines change quickly: flags, defaults and supported models move between releases.
Check yourself
A server's throughput is flat as the request rate rises, but TTFT keeps growing. What is happening?
Check yourself
Why does Llama 3 8B fit far more concurrent sequences than Llama 2 13B on the same GPU?
Remember
- ✓A serving engine does API, tokenization, per-step scheduling, paged KV memory, batched forward passes, sampling and streaming.
- ✓Latency ; concurrency (Little's law) sizes the KV memory you need.
- ✓KV pool = utilisation × GPU memory − weights − activations; it caps the batch and hence throughput.
- ✓Throughput rises with load until compute, bandwidth or KV memory saturates; then only the queue (TTFT) grows.
- ✓vLLM: Python engine, PagedAttention, OpenAI API on port 8000. TGI: Rust router + Python shards, native and OpenAI APIs.