# Inference servers (vLLM, TGI)

> An LLM inference server such as vLLM or TGI tokenizes requests, batches them continuously, packs a paged KV cache into spare GPU memory, runs the model and streams tokens, trading throughput against time to first token and per-token latency.

Source: https://www.mlfoundry.in/learn/inference-servers  
Section: Generative AI › Serving Generative Models  
Level: core · 8 min read · updated 2026-10-21

**Inference server:** (serving engine) is the long-running process that holds a model's weights on accelerators and turns API requests into generated tokens. It owns tokenization, scheduling and batching, KV-cache memory, the forward pass, sampling and streaming. **vLLM** came from UC Berkeley (open-sourced June 2023; Kwon et al., SOSP 2023). **Text Generation Inference (TGI)** is Hugging Face's server, released in 2023. Other widely used engines are NVIDIA TensorRT-LLM, SGLang (Zheng et al., 2024) and, for local use, llama.cpp's server and Ollama.

Calling `model.generate()` inside a web handler works for a demo. Under real traffic it serves one request or one fixed batch at a time, reserves cache memory for the maximum length, and leaves most of the GPU idle while it waits for the longest answer. A serving engine adds the systems layer: **Continuous batching** to keep the batch full, **PagedAttention** to pack the **KV cache**, fused kernels, quantized weights, multi-GPU sharding and token streaming. In a product it sits behind a gateway that handles auth, rate limits and routing, and in front of the GPUs.

## Request path

1. **API server.** Accepts an OpenAI-style `/v1/chat/completions` (or a native) request, validates it, applies the model's chat template and tokenizes.
2. **Scheduler.** Before every forward pass, decides which sequences run: running ones continue, finished ones leave, waiting ones are admitted while slots and KV blocks remain.
3. **KV block manager.** Hands out cache blocks, reuses cached prefixes, and preempts sequences when memory runs out.
4. **Model executor.** Runs one forward pass over the whole batch on one or more GPUs (tensor-parallel shards), using attention kernels that read the paged cache.
5. **Sampler and detokenizer.** Applies temperature, top-p and the other sampling parameters, checks stop conditions and streams text deltas back as server-sent events (see **Streaming generation**).

> Figure: A streaming chat request through a serving engine. The API server tokenizes; the scheduler and KV block manager choose the running batch each step (3,750 blocks of 16 tokens is the pool of Llama 2 13B at 0.9 memory utilisation on an 80 GiB GPU); the executor runs one pass for all sequences; the sampler emits one token per sequence, which is streamed to the client as an SSE chunk and fed back for the next step. vLLM runs this as a Python engine; TGI splits it into a Rust router and Python model shards.

## What to measure

$$
T_\text{e2e}=\text{TTFT}+(n_\text{out}-1)\cdot\text{TPOT},\qquad L=\lambda\,T_\text{e2e}
$$

- $\text{TTFT}$: time to first token: queueing + prefill of the prompt
- $\text{TPOT}$: time per output token after the first (inter-token latency), set by the decode step time
- $L,\ \lambda$: Little's law: sequences in the system = arrival rate × time each spends there

Throughput (output tokens per second across all users) is the cost metric. TTFT and TPOT are the user-facing ones. They pull against each other: a larger batch raises throughput, but every sequence's step gets slower. *Goodput* counts only requests that meet both latency targets, and it is the number to optimise against.

$$
N_\text{KV}=\left\lfloor\frac{u\cdot M_\text{GPU}-M_\text{weights}-M_\text{act}}{2\,L_\text{layers}\,H_\text{kv}\,d_\text{head}\,\text{bytes}}\right\rfloor,\qquad t_\text{step}\gtrsim\frac{M_\text{weights}+\sum_i n_i\cdot\text{kv}}{\text{HBM bandwidth}}
$$

- $u$: fraction of GPU memory the engine may use (vLLM `--gpu-memory-utilization`, default 0.9)
- $M_\text{act}$: activation and workspace memory, measured at start-up by a profiling forward pass
- $N_\text{KV}$: tokens the KV pool can hold across all running sequences
- $t_\text{step}$: decode step time: weights plus every live sequence's cache ($n_i$ tokens, kv bytes each) read once per step

vLLM and TGI in brief (both move quickly; check current docs before relying on a flag)

| Aspect | vLLM | TGI |
| --- | --- | --- |
| Origin | UC Berkeley, 2023 (PagedAttention paper, SOSP 2023) | Hugging Face, 2023 |
| Structure | Python engine on PyTorch with custom CUDA kernels | Rust launcher and router (HTTP, batching) + Python model shards over gRPC |
| Start | `vllm serve <model>` (port 8000) | Docker image with `--model-id <model>` (port 80 in the container) |
| API | OpenAI-compatible `/v1/completions`, `/v1/chat/completions` | native `/generate`, `/generate_stream`, and an OpenAI-compatible `/v1/chat/completions` |
| Batching and memory | continuous batching, PagedAttention, prefix caching, chunked prefill | continuous batching, FlashAttention and paged attention kernels |
| Multi-GPU | `--tensor-parallel-size`, `--pipeline-parallel-size` | `--num-shard` (tensor parallel) |
| Quantization | GPTQ, AWQ, FP8 and others; `--kv-cache-dtype fp8` | `--quantize` with bitsandbytes, GPTQ, AWQ, EETQ, FP8 and others |
| Licence | Apache 2.0 | Apache 2.0 (HFOIL 1.0 for v1.0, July 2023; Apache again from v2.0, April 2024) |

The knobs that matter most, and what they trade

| Knob (vLLM / TGI) | Raises | Costs |
| --- | --- | --- |
| `--gpu-memory-utilization` / `--cuda-memory-fraction` | KV pool, so batch size and throughput | less headroom; out-of-memory risk with other processes on the GPU |
| `--max-num-seqs` / `--max-batch-total-tokens` | concurrency and throughput | TPOT grows with every added sequence |
| `--max-num-batched-tokens` / `--max-batch-prefill-tokens` | prefill throughput per step | longer steps stall the decoding sequences |
| `--max-model-len` / `--max-total-tokens` | longest request served | fewer long requests fit at once |
| quantized weights or fp8 KV cache | free memory and bandwidth, so batch and speed | some accuracy; evaluate on your task (see **Quantization**) |

- **vLLM:** general GPU serving with wide model coverage and an OpenAI-compatible API; also used as the generation engine inside RL and evaluation pipelines
- **TGI:** Hugging Face deployments (Inference Endpoints, the Hub's model formats) with a Rust front end and Prometheus metrics
- **TensorRT-LLM:** NVIDIA GPUs, with engines compiled ahead of time per model and GPU, usually behind Triton Inference Server
- **SGLang:** workloads with heavy prefix reuse and structured, multi-call programs (RadixAttention keeps a tree of cached prefixes)
- **llama.cpp server, Ollama:** quantized GGUF models on CPUs, Apple silicon and small GPUs, mostly single-user (see **On-device generative AI**)

> Figure: A simulated server on one 80 GiB GPU (2.0 TB/s, prefill at 156 TFLOPS) with 200 requests: prompts of 100–899 tokens, mean output 208 tokens. Defaults (Llama 2 13B, utilisation 0.9, max_num_seqs 256): the pool holds 60,012 tokens; at 6 req/s the server gives 848 tok/s, TTFT 0.18 s (p90 0.29 s), TPOT 38.6 ms and peak KV use 61%. At 10 req/s throughput saturates at 925 tok/s, the pool fills, 25 sequences are preempted, TPOT is 73.8 ms and p90 TTFT 5.46 s. max_num_seqs 32 at 10 req/s keeps TPOT at 32.9 ms but gives 803 tok/s and p90 TTFT 15.49 s. Llama 3 8B (GQA, 128 KiB per token) reaches 1,890 tok/s at 16 req/s using 15% of its pool.

Two walls appear in the figure. Up to about 8 req/s the server keeps up: throughput grows with load and TTFT stays near the prefill time. Past that, the GPU is saturated. Extra requests only lengthen the queue, so TTFT explodes while throughput flattens. For the 13B model the wall is memory: its 800 KiB per token fills the pool and forces preemption. Lowering utilisation to 0.6 cuts the pool to 28,554 tokens and raises preemptions at 10 req/s from 25 to 58.

## Worked example: sizing one GPU

1. **Pool.** Llama 2 13B in bf16 is $2\times13.0\text{B}=26.0$ GB = 24.21 GiB. With $u=0.9$ of 80 GiB and 2 GiB of activations: $72-24.21-2=45.79$ GiB. At 800 KiB per token that is 60,012 tokens, or 3,750 blocks of 16.
2. **Target load.** 5 req/s, mean output 208 tokens, TTFT 0.2 s and TPOT 40 ms. Then $T_\text{e2e}=0.2+207\times0.04=8.48$ s, and Little's law gives $L=5\times8.48=42.4$ sequences in flight.
3. **Memory.** Mean context per sequence is the prompt plus half the output: $484+104=588$ tokens (459 MiB). In total $42.4\times588=24{,}931$ tokens, 41.5% of the pool, so it fits.
4. **Speed.** Pure decode reads $26.0+24{,}931\times819{,}200/10^9=46.4$ GB per step, 23.2 ms at 2 TB/s. That is inside the 40 ms TPOT budget, which leaves time for prefills. The demand of $5\times208=1{,}040$ tok/s is below the ideal decode rate of $42.4/0.0232=1{,}827$ tok/s.

```python
# vLLM:  vllm serve meta-llama/Llama-2-13b-chat-hf --gpu-memory-utilization 0.9 --max-num-seqs 256
# TGI:   docker run --gpus all -p 8080:80 ghcr.io/huggingface/text-generation-inference:<version> \
#            --model-id meta-llama/Llama-2-13b-chat-hf        (then use base_url http://localhost:8080/v1)
import time
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
t0, first, chunks = time.perf_counter(), None, 0
stream = client.chat.completions.create(
    model="meta-llama/Llama-2-13b-chat-hf",
    messages=[{"role": "user", "content": "Explain the KV cache in two sentences."}],
    max_tokens=256, temperature=0.7, stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        if first is None:
            first = time.perf_counter() - t0          # TTFT
        chunks += 1                                   # usually one token per chunk
total = time.perf_counter() - t0
print(f"TTFT {first:.3f} s, TPOT {1000 * (total - first) / max(chunks - 1, 1):.1f} ms, {chunks} chunks")
```

Launching vLLM or TGI and measuring TTFT and TPOT from a streaming OpenAI-compatible request (needs a GPU server and the openai package; not run here)

> **Benchmark the traffic you will get:** Engine comparisons swing with prompt and output lengths, request rate, sampling settings and hardware. A benchmark that sends all requests at once measures peak throughput. One that sends them one at a time measures single-user latency. Neither predicts a product. Replay a trace with realistic lengths at several arrival rates, as in the figure, and report throughput together with TTFT and TPOT percentiles. Vendor numbers are dated: the June 2023 vLLM launch post claimed up to 24× the throughput of Hugging Face Transformers and up to 3.5× that of TGI at the time. Both engines have changed a great deal since.

> **Throughput tricks live in the engine:** Most of what **Cost and latency optimization** can do on the serving side is engine configuration: prefix caching for shared system prompts (**Prompt caching**), **Speculative decoding** for latency at low load, fp8 KV caches and **Quantization** for memory, and tensor parallelism when the model or its cache does not fit one GPU.

**Strengths**

- Self-hosting an engine gives control of the model, data locality, and cost per token at high, steady utilisation.
- One OpenAI-compatible endpoint lets applications switch between hosted APIs and your own models.
- Engine features (prefix caching, quantization, speculative decoding) are configuration, not application code.

**Limitations**

- GPUs cost the same idle or busy; at low or bursty traffic a hosted API is often cheaper.
- Latency targets, memory headroom and preemption need tuning and monitoring per model and GPU.
- Engines change quickly: flags, defaults and supported models move between releases.

**Question:** A server's throughput is flat as the request rate rises, but TTFT keeps growing. What is happening?

- A. The tokenizer is the bottleneck
- B. The GPU is saturated (compute, bandwidth or KV memory), so extra requests only wait in the queue
- C. Continuous batching is turned off
- D. TPOT is falling, which delays the first token

**Answer:** B. The GPU is saturated (compute, bandwidth or KV memory), so extra requests only wait in the queue — Past saturation, every step is already as full as memory and max_num_seqs allow. New arrivals join the waiting queue, so their time to first token grows with the queue length while output tokens per second stay constant.

**Question:** Why does Llama 3 8B fit far more concurrent sequences than Llama 2 13B on the same GPU?

- A. It has fewer layers
- B. Its weights are smaller and grouped-query attention cuts the KV cache to 128 KiB per token, against 800 KiB
- C. vLLM compresses its cache automatically
- D. It uses a smaller vocabulary

**Answer:** B. Its weights are smaller and grouped-query attention cuts the KV cache to 128 KiB per token, against 800 KiB — Both layer counts are similar (32 vs 40). The big difference is 8 KV heads instead of 40: $2\cdot32\cdot8\cdot128\cdot2=131{,}072$ bytes per token against $2\cdot40\cdot40\cdot128\cdot2=819{,}200$. Together with 10 GB less of weights, the pool holds 450,911 tokens instead of 60,012.

**Key points**

- A serving engine does API, tokenization, per-step scheduling, paged KV memory, batched forward passes, sampling and streaming.
- Latency $=\text{TTFT}+(n-1)\,\text{TPOT}$; concurrency $L=\lambda T$ (Little's law) sizes the KV memory you need.
- KV pool = utilisation × GPU memory − weights − activations; it caps the batch and hence throughput.
- Throughput rises with load until compute, bandwidth or KV memory saturates; then only the queue (TTFT) grows.
- vLLM: Python engine, PagedAttention, OpenAI API on port 8000. TGI: Rust router + Python shards, native and OpenAI APIs.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
