# Cost and latency optimization

> An LLM request costs N_in·p_in + N_out·p_out and takes TTFT + (N_out − 1)·TPOT; most savings come from fewer output tokens, cached prefixes, routing easy requests to small models, batching and quantization.

Source: https://www.mlfoundry.in/learn/cost-and-latency-optimization  
Section: Generative AI › Serving Generative Models  
Level: core · 8 min read · updated 2026-10-21

**Cost and latency optimization:** is the engineering of an LLM application so that each request uses fewer paid tokens and fewer GPU-seconds, and returns sooner, while quality stays above a fixed bar. The main levers are prompt length, output length, caching, model choice and routing (FrugalGPT, Chen, Zaharia and Zou, 2023), and serving-side techniques such as batching, quantization and speculative decoding.

A chat or RAG product pays for every token twice: once in money, through per-token API prices or GPU rental, and once in time, through the wait the user sees. Both scale with traffic, so a design that is fine for a demo can cost thousands of dollars a day in production. You optimise against a quality metric from your own evaluation set. Cutting cost by moving to a model that fails a third of your cases is not an optimisation.

Two different latencies matter. **Time to first token (TTFT)** is how long the user waits before anything appears. It is dominated by queueing and by *prefill*, the parallel pass over the whole prompt. **Time per output token (TPOT)**, also called inter-token latency, is the step time of *decode*, which generates one token at a time. With **Streaming generation** the user perceives TTFT. A program that needs the full JSON perceives the total.

## The two formulas

$$
C=\frac{(N_\text{in}-N_\text{c})\,p_\text{in}+N_\text{c}\,\alpha\,p_\text{in}+N_\text{out}\,p_\text{out}}{10^{6}}
$$

- $N_\text{in},N_\text{out}$: input and output tokens of the request
- $N_\text{c}$: input tokens served from a prompt cache
- $\alpha$: price multiplier for cached input, well below 1 (0.1 in this article's examples)
- $p_\text{in},p_\text{out}$: price per million tokens; output is usually several times dearer than input

$$
T=\underbrace{t_\text{queue}+t_0+\frac{N_\text{in}-N_\text{c}}{v_\text{prefill}}}_{\text{TTFT}}+(N_\text{out}-1)\cdot\text{TPOT},\qquad \text{TPOT}\gtrsim\frac{\text{bytes of weights}+\text{bytes of KV cache}}{\text{memory bandwidth}}
$$

- $v_\text{prefill}$: prompt tokens processed per second; prefill is compute-bound
- $\text{TPOT}$: decode step time; at small batch sizes it is memory-bandwidth-bound

The second bound explains much of serving. An 8B-parameter model in bf16 is 16 GB of weights. An H100 SXM reads about 3.35 TB/s from HBM, so one decode step cannot take less than about 4.8 ms, or about 209 tokens/s for a single sequence. A batch of 32 sequences reads the same weights once per step, so throughput rises almost 32-fold until compute or KV-cache reads take over. That is why **Continuous batching** and **Quantization** (fewer bytes per weight) pay off so well.

> Figure: One request with a 1,600-token static prefix, a 400-token user turn and a 300-token answer on a large model (illustrative $3 / $15 per million input/output tokens, 0.25 s overhead, 8,000 tokens/s prefill, 25 ms per output token). TTFT is 0.50 s and decode adds 7.48 s, so 94% of the 7.98 s wait is generation. Caching the prefix cuts the cost from $0.0105 to $0.00618; also capping the answer at 150 tokens brings it to $0.00393 and 4.03 s.

## The levers, in order of usual payoff

1. **Measure first.** Log input, cached and output tokens, TTFT and total latency per request type. Find the endpoint that dominates spend before you touch anything.
2. **Cut output tokens.** Ask for terse formats, set `max_tokens`, use stop sequences, and return structured fields rather than prose. Output tokens cost the most and each one adds a decode step.
3. **Make prompts cacheable.** Put the system prompt, tool schemas and shared documents first and the variable part last, so **Prompt caching** can reuse the prefix. For repeated questions, a **Semantic caching** hit skips the model entirely.
4. **Shrink the input.** Retrieve fewer, better chunks (see **Reranking**), drop stale chat turns, and compress long contexts. LLMLingua (Jiang et al., 2023) drops low-information tokens and reports up to 20× compression with small quality loss.
5. **Route or cascade.** Send easy requests to a small model. A router predicts difficulty up front. A cascade tries the small model and escalates when a scorer is not confident (see **Model families and model selection**).
6. **Move work off the critical path.** Non-interactive jobs (evaluation, back-fills, nightly summaries) go to a batch endpoint at a discount.
7. **If you self-host, tune the server.** Use continuous batching and **PagedAttention** (see **Inference servers (vLLM, TGI)**), weight and KV-cache quantization, **Speculative decoding** for latency, and **Knowledge distillation** into a smaller model for a narrow task.

**Checkable specifics (as of late 2024; prices and policies change)**

- **Output vs input price:** GPT-4o listed USD 2.50 / 10 and Claude 3.5 Sonnet USD 3 / 15 per million input/output tokens: output 4–5× dearer
- **Batch APIs:** OpenAI's Batch API (April 2024) and Anthropic's Message Batches API (October 2024) both charge 50% of the normal price, with results returned asynchronously, within 24 hours
- **Prompt caching:** OpenAI caches prompts of 1,024+ tokens automatically, with cached input at 50% off. Anthropic bills cache reads at 10% of the input price and cache writes at 125%
- **FrugalGPT:** an LLM cascade with a learned scorer; the paper reports matching the best single model's accuracy at up to 98% lower cost on its benchmarks
- **Decode bound:** 8B params × 2 bytes ÷ 3.35 TB/s ≈ 4.8 ms per step at batch 1 on one H100 SXM; a 4-bit copy reads a quarter of the bytes

## Simulating routing and caching

The figure below simulates 1,000 requests with difficulty $d\sim U(0,1)$. The large model answers correctly with probability $\sigma((0.95-d)/0.05)$, the small one with $\sigma((0.55-d)/0.08)$. The router sees a noisy estimate $\hat d=d+0.12\,\varepsilon$ and sends the request to the small model when $\hat d<\tau$. The cascade always runs the small model first, then escalates when a noisy confidence score falls below $\tau$, paying for both calls.

> Figure: Defaults (router, τ = 0.40, 2,000 in / 300 out, no cache): 40.2% of requests go to the small model; accuracy 91.5% vs 93.5% for large-only, $6.47 vs $10.50 per 1,000 requests, mean latency 5.80 s vs 7.98 s. At τ = 0.30 the router keeps 93.0% at $7.46. The cascade at τ = 0.70 reaches 92.5% for $6.92 but escalates 61.3% of requests, so p95 latency rises to 10.53 s. Set cache to 80% and output to 150: the router costs $2.42 with mean latency 2.93 s.

Routing and caching act on different terms of the formulas, so their savings multiply. A cascade spends money on every escalated request and adds the small model's full latency to it. It is the right tool when the small model handles most of the traffic and its mistakes can be detected cheaply. A router decides before any generation, so it never pays twice, but it must predict difficulty from the prompt alone.

## Worked example

One request type, 2,000 input / 300 output tokens, large model at USD 3 / 15 per 1M (illustrative), cache reads at 10%. The last row is the simulation's mean.

| Step | Input cost | Output cost | Total / request | Latency |
| --- | --- | --- | --- | --- |
| Baseline | 2,000 × $3/1M = $0.00600 | 300 × $15/1M = $0.00450 | $0.01050 | 0.50 + 299 × 0.025 = 7.98 s |
| + cache 1,600-token prefix | 400 × $3/1M + 1,600 × $0.30/1M = $0.00168 | $0.00450 | $0.00618 (−41%) | 0.30 + 7.48 = 7.78 s |
| + cap answer at 150 tokens | $0.00168 | 150 × $15/1M = $0.00225 | $0.00393 (−63%) | 0.30 + 149 × 0.025 = 4.03 s |
| + router, τ = 0.40 | mixed | mixed | $0.00242 (−77%) | mean 2.93 s, p95 4.03 s |

At one million requests a month the baseline costs $10,500 and the last row $2,420, at a cost of two accuracy points on this toy distribution. Whether that trade is acceptable is a product decision. Make it on your own evaluation set, not on a benchmark.

```python
PRICES = {  # $ per 1M tokens: (input, cached input, output); illustrative
    "large": (3.00, 0.30, 15.00),
    "small": (0.15, 0.015, 0.60),
}

def request_cost(model, usage):
    p_in, p_cached, p_out = PRICES[model]
    cached = usage.get("cached_tokens", 0)
    fresh = usage["input_tokens"] - cached
    return (fresh * p_in + cached * p_cached + usage["output_tokens"] * p_out) / 1e6

def latency(ttft_s, tpot_s, out_tokens):
    return ttft_s + (out_tokens - 1) * tpot_s

def cascade(question, small, large, judge, tau=0.7):
    """Small model first; escalate to the large one when the judge is not confident."""
    draft, u1 = small(question)
    if judge(question, draft) >= tau:
        return draft, request_cost("small", u1)
    final, u2 = large(question)
    return final, request_cost("small", u1) + request_cost("large", u2)  # paid twice

usage = {"input_tokens": 2000, "cached_tokens": 1600, "output_tokens": 150}
print(round(request_cost("large", usage), 5))   # 0.00393
print(round(request_cost("small", usage), 6))   # 0.000174
print(round(latency(0.30, 0.025, 150), 3))      # 4.025
```

Per-request cost from usage counts, and a small-first cascade (runs as is). Real APIs report cached tokens in fields such as usage.prompt_tokens_details.cached_tokens (OpenAI) or usage.cache_read_input_tokens (Anthropic).

> **Optimising the mean, shipping the tail:** Users notice p95 and p99 latency, not the mean. A cascade lowers mean cost but makes its escalated requests the slowest ones. A long-context request can block a batch slot and raise everyone's TTFT. Report percentiles per request type. Set separate budgets for TTFT and for total time.

> **Throughput and latency pull in opposite directions:** On self-hosted GPUs, a bigger batch lowers cost per token, because the weights are read once for many sequences. But it raises each sequence's TPOT once compute saturates, and it adds queueing. Serving systems expose this as a knob: maximum batch size or tokens per step. Pick it from a latency SLO, then maximise throughput under it.

**Strengths**

- Output limits, prompt order for caching and batch endpoints need no model change and keep quality intact.
- Routing and caching compound with each other and with lower token prices.
- Self-hosted quantization and batching cut GPU-hours by large factors on bandwidth-bound decode.

**Limitations**

- Routers and cascades add a component that can silently misroute hard requests; they need their own evaluation.
- Aggressive prompt compression or chunk pruning can remove the one fact the answer needed.
- Every cache adds staleness and invalidation problems; semantic caches can return a wrong answer to a similar-looking question.

**Question:** A request has 3,000 input tokens and 600 output tokens; prefill runs at 10,000 tokens/s with 0.2 s overhead and TPOT is 20 ms. What dominates its latency?

- A. Prefill: about 0.5 s of the total
- B. Decode: about 12 s of the total
- C. They are about equal
- D. Network transfer of the prompt

**Answer:** B. Decode: about 12 s of the total — TTFT = 0.2 + 3,000/10,000 = 0.5 s, and decode = 599 × 0.02 = 11.98 s, so the total is 12.48 s and 96% of it is decode. Shortening the answer saves far more time than shortening the prompt.

**Question:** Why does a batch of 16 sequences decode almost 16 times more tokens per second than one sequence on the same GPU, at nearly the same TPOT?

- A. Batching shrinks the KV cache
- B. At small batch sizes each decode step is limited by reading the weights from memory, and one read serves every sequence in the batch
- C. The GPU raises its clock speed under load
- D. Batched sequences share output tokens

**Answer:** B. At small batch sizes each decode step is limited by reading the weights from memory, and one read serves every sequence in the batch — A decode step is a matrix–vector product per sequence. Reading 16 GB of weights costs the same whether one or 16 vectors are multiplied by them. Gains stop when the step becomes compute-bound or when per-sequence KV-cache reads dominate.

**Key points**

- Cost $=\big((N_\text{in}-N_\text{c})p_\text{in}+N_\text{c}\alpha p_\text{in}+N_\text{out}p_\text{out}\big)/10^6$; latency $=$ TTFT $+(N_\text{out}-1)\cdot$TPOT.
- Output tokens are the dearest and the slowest; cutting them is usually the first and biggest win.
- Static content first makes the prefix cacheable: cheaper input and a shorter prefill.
- Routers pick a model up front; cascades try small first and pay twice on escalation, which hurts tail latency.
- Decode is memory-bandwidth-bound at small batch: batching and quantization raise throughput almost for free.
- Optimise against your own quality bar and report latency percentiles, not means.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
