Cost and latency optimization

An LLM request costs N_in·p_in + N_out·p_out and takes TTFT + (N_out − 1)·TPOT; most savings come from fewer output tokens, cached prefixes, routing easy requests to small models, batching and quantization.

8 min read881 wordscore0 viewsupdated 2026-10-21

Definition

Cost and latency optimization is the engineering of an LLM application so that each request uses fewer paid tokens and fewer GPU-seconds, and returns sooner, while quality stays above a fixed bar. The main levers are prompt length, output length, caching, model choice and routing (FrugalGPT, Chen, Zaharia and Zou, 2023), and serving-side techniques such as batching, quantization and speculative decoding.

A chat or RAG product pays for every token twice: once in money, through per-token API prices or GPU rental, and once in time, through the wait the user sees. Both scale with traffic, so a design that is fine for a demo can cost thousands of dollars a day in production. You optimise against a quality metric from your own evaluation set. Cutting cost by moving to a model that fails a third of your cases is not an optimisation.

Two different latencies matter. Time to first token (TTFT) is how long the user waits before anything appears. It is dominated by queueing and by prefill, the parallel pass over the whole prompt. Time per output token (TPOT), also called inter-token latency, is the step time of decode, which generates one token at a time. With Streaming generation the user perceives TTFT. A program that needs the full JSON perceives the total.

The two formulas

C=(Nin−Nc) pin+Nc α pin+Nout pout106C=\frac{(N_\text{in}-N_\text{c})\,p_\text{in}+N_\text{c}\,\alpha\,p_\text{in}+N_\text{out}\,p_\text{out}}{10^{6}}
Nin,NoutN_\text{in},N_\text{out}
input and output tokens of the request
NcN_\text{c}
input tokens served from a prompt cache
α\alpha
price multiplier for cached input, well below 1 (0.1 in this article's examples)
pin,poutp_\text{in},p_\text{out}
price per million tokens; output is usually several times dearer than input
T=tqueue+t0+Nin−Ncvprefill⏟TTFT+(Nout−1)⋅TPOT,TPOT≳bytes of weights+bytes of KV cachememory bandwidthT=\underbrace{t_\text{queue}+t_0+\frac{N_\text{in}-N_\text{c}}{v_\text{prefill}}}_{\text{TTFT}}+(N_\text{out}-1)\cdot\text{TPOT},\qquad \text{TPOT}\gtrsim\frac{\text{bytes of weights}+\text{bytes of KV cache}}{\text{memory bandwidth}}
vprefillv_\text{prefill}
prompt tokens processed per second; prefill is compute-bound
TPOT\text{TPOT}
decode step time; at small batch sizes it is memory-bandwidth-bound

The second bound explains much of serving. An 8B-parameter model in bf16 is 16 GB of weights. An H100 SXM reads about 3.35 TB/s from HBM, so one decode step cannot take less than about 4.8 ms, or about 209 tokens/s for a single sequence. A batch of 32 sequences reads the same weights once per step, so throughput rises almost 32-fold until compute or KV-cache reads take over. That is why Continuous batching and Quantization (fewer bytes per weight) pay off so well.

one request through the stack, and the lever at each stage1 prompt1,600 tok prefix(system + docs)+ 400 tok user2 cache / routesemantic cache?easy → smallhard → large3 prefillall 2,000 tokensin parallel;sets TTFT4 decode1 token / step,weights + KVread each step5 response300 output tokstreamed;max_tokens capleversstatic text first→ prompt cachingtrim / compress(LLMLingua)cache hit: nomodel call at allrouter / cascade(FrugalGPT)cached prefixskips prefillchunked prefillfor long inputsquantize weightsspeculative dec.batch requests(throughput)shorter answersstop sequencesBatch API: 50%off, asynclatency, large model (TTFT = 0.25 s + 2,000 / 8,000 tok/s; 25 ms per output token)TTFT 0.50 s299 more tokens × 25 ms = 7.48 s of decode7.98 sdecode is 94% of the wait: output length is the biggest latency levercost at $3 / $15 per 1M input / output tokens (illustrative)input 2,000 × $3/1M = $0.0060output 300 × $15/1M = $0.0045total $0.0105 per request = $10.50 per 1,000 requestscache the 1,600-token prefix (read at 10%): input → $0.00168, total → $0.00618 (−41%)and cap the answer at 150 tokens: total → $0.00393 (−63%), latency → 0.30 + 3.73 = 4.03 s
fig.One request with a 1,600-token static prefix, a 400-token user turn and a 300-token answer on a large model (illustrative $3 / $15 per million input/output tokens, 0.25 s overhead, 8,000 tokens/s prefill, 25 ms per output token). TTFT is 0.50 s and decode adds 7.48 s, so 94% of the 7.98 s wait is generation. Caching the prefix cuts the cost from $0.0105 to $0.00618; also capping the answer at 150 tokens brings it to $0.00393 and 4.03 s.

The levers, in order of usual payoff

  1. 1Measure first. Log input, cached and output tokens, TTFT and total latency per request type. Find the endpoint that dominates spend before you touch anything.
  2. 2Cut output tokens. Ask for terse formats, set max_tokens, use stop sequences, and return structured fields rather than prose. Output tokens cost the most and each one adds a decode step.
  3. 3Make prompts cacheable. Put the system prompt, tool schemas and shared documents first and the variable part last, so Prompt caching can reuse the prefix. For repeated questions, a Semantic caching hit skips the model entirely.
  4. 4Shrink the input. Retrieve fewer, better chunks (see Reranking), drop stale chat turns, and compress long contexts. LLMLingua (Jiang et al., 2023) drops low-information tokens and reports up to 20× compression with small quality loss.
  5. 5Route or cascade. Send easy requests to a small model. A router predicts difficulty up front. A cascade tries the small model and escalates when a scorer is not confident (see Model families and model selection).
  6. 6Move work off the critical path. Non-interactive jobs (evaluation, back-fills, nightly summaries) go to a batch endpoint at a discount.
  7. 7If you self-host, tune the server. Use continuous batching and PagedAttention (see Inference servers (vLLM, TGI)), weight and KV-cache quantization, Speculative decoding for latency, and Knowledge distillation into a smaller model for a narrow task.

Checkable specifics (as of late 2024; prices and policies change)

Output vs input price
GPT-4o listed USD 2.50 / 10 and Claude 3.5 Sonnet USD 3 / 15 per million input/output tokens: output 4–5× dearer
Batch APIs
OpenAI's Batch API (April 2024) and Anthropic's Message Batches API (October 2024) both charge 50% of the normal price, with results returned asynchronously, within 24 hours
Prompt caching
OpenAI caches prompts of 1,024+ tokens automatically, with cached input at 50% off. Anthropic bills cache reads at 10% of the input price and cache writes at 125%
FrugalGPT
an LLM cascade with a learned scorer; the paper reports matching the best single model's accuracy at up to 98% lower cost on its benchmarks
Decode bound
8B params × 2 bytes ÷ 3.35 TB/s ≈ 4.8 ms per step at batch 1 on one H100 SXM; a 4-bit copy reads a quarter of the bytes

Simulating routing and caching

The figure below simulates 1,000 requests with difficulty d∼U(0,1)d\sim U(0,1). The large model answers correctly with probability σ((0.95−d)/0.05)\sigma((0.95-d)/0.05), the small one with σ((0.55−d)/0.08)\sigma((0.55-d)/0.08). The router sees a noisy estimate d^=d+0.12 ε\hat d=d+0.12\,\varepsilon and sends the request to the small model when d^<τ\hat d<\tau. The cascade always runs the small model first, then escalates when a noisy confidence score falls below τ\tau, paying for both calls.

1000 requests · 2000 in + 300 out tokens · prefix cached 0%accuracy vs $ per 1k requests (all thresholds)0.50.60.70.80.91.00.02.85.68.411large onlyroutercascadecurrentlatency per request, slarge, no cacheTTFT 0.50 + gen 7.48 · p95 7.98large + this cacheTTFT 0.50 + gen 7.48 · p95 7.98current setupTTFT 0.37 + gen 5.43 · p95 7.98solid TTFT · light generation · | p95current: accuracy 91.5% · $6.47 per 1k · mean 5.80 s · p95 7.98 svs large only, no cache: 93.5% · $10.50 · mean 7.98 s → cost ×0.62, accuracy −2.0 ptsanswered by small model 40.2%large: $3/$15 per 1M in/out, 25 ms/token, prefill 8k tok/s · small: $0.15/$0.6, 8 ms/token, 30k tok/scached input billed at 10% and skips prefill · prices and speeds are illustrative, not vendor list pricesrouter: noisy difficulty estimate d̂ = d + 0.12·ε · cascade: small answers, low confidence → also call large
serving
fig.Defaults (router, τ = 0.40, 2,000 in / 300 out, no cache): 40.2% of requests go to the small model; accuracy 91.5% vs 93.5% for large-only, $6.47 vs $10.50 per 1,000 requests, mean latency 5.80 s vs 7.98 s. At τ = 0.30 the router keeps 93.0% at $7.46. The cascade at τ = 0.70 reaches 92.5% for $6.92 but escalates 61.3% of requests, so p95 latency rises to 10.53 s. Set cache to 80% and output to 150: the router costs $2.42 with mean latency 2.93 s.

Routing and caching act on different terms of the formulas, so their savings multiply. A cascade spends money on every escalated request and adds the small model's full latency to it. It is the right tool when the small model handles most of the traffic and its mistakes can be detected cheaply. A router decides before any generation, so it never pays twice, but it must predict difficulty from the prompt alone.

Worked example

StepInput costOutput costTotal / requestLatency
Baseline2,000 × 3/1M=3/1M = 0.00600300 × 15/1M=15/1M = 0.00450$0.010500.50 + 299 × 0.025 = 7.98 s
+ cache 1,600-token prefix400 × 3/1M+1,600×3/1M + 1,600 × 0.30/1M = $0.00168$0.00450$0.00618 (−41%)0.30 + 7.48 = 7.78 s
+ cap answer at 150 tokens$0.00168150 × 15/1M=15/1M = 0.00225$0.00393 (−63%)0.30 + 149 × 0.025 = 4.03 s
+ router, τ = 0.40mixedmixed$0.00242 (−77%)mean 2.93 s, p95 4.03 s
One request type, 2,000 input / 300 output tokens, large model at USD 3 / 15 per 1M (illustrative), cache reads at 10%. The last row is the simulation's mean.

At one million requests a month the baseline costs 10,500andthelastrow10,500 and the last row 2,420, at a cost of two accuracy points on this toy distribution. Whether that trade is acceptable is a product decision. Make it on your own evaluation set, not on a benchmark.

python
1PRICES = { # $ per 1M tokens: (input, cached input, output); illustrative
2 "large": (3.00, 0.30, 15.00),
3 "small": (0.15, 0.015, 0.60),
4}
5
6def request_cost(model, usage):
7 p_in, p_cached, p_out = PRICES[model]
8 cached = usage.get("cached_tokens", 0)
9 fresh = usage["input_tokens"] - cached
10 return (fresh * p_in + cached * p_cached + usage["output_tokens"] * p_out) / 1e6
11
12def latency(ttft_s, tpot_s, out_tokens):
13 return ttft_s + (out_tokens - 1) * tpot_s
14
15def cascade(question, small, large, judge, tau=0.7):
16 """Small model first; escalate to the large one when the judge is not confident."""
17 draft, u1 = small(question)
18 if judge(question, draft) >= tau:
19 return draft, request_cost("small", u1)
20 final, u2 = large(question)
21 return final, request_cost("small", u1) + request_cost("large", u2) # paid twice
22
23usage = {"input_tokens": 2000, "cached_tokens": 1600, "output_tokens": 150}
24print(round(request_cost("large", usage), 5)) # 0.00393
25print(round(request_cost("small", usage), 6)) # 0.000174
26print(round(latency(0.30, 0.025, 150), 3)) # 4.025
Per-request cost from usage counts, and a small-first cascade (runs as is). Real APIs report cached tokens in fields such as usage.prompt_tokens_details.cached_tokens (OpenAI) or usage.cache_read_input_tokens (Anthropic).

Use when

  • +Output limits, prompt order for caching and batch endpoints need no model change and keep quality intact.
  • +Routing and caching compound with each other and with lower token prices.
  • +Self-hosted quantization and batching cut GPU-hours by large factors on bandwidth-bound decode.

Avoid when

  • −Routers and cascades add a component that can silently misroute hard requests; they need their own evaluation.
  • −Aggressive prompt compression or chunk pruning can remove the one fact the answer needed.
  • −Every cache adds staleness and invalidation problems; semantic caches can return a wrong answer to a similar-looking question.

Check yourself

A request has 3,000 input tokens and 600 output tokens; prefill runs at 10,000 tokens/s with 0.2 s overhead and TPOT is 20 ms. What dominates its latency?

Check yourself

Why does a batch of 16 sequences decode almost 16 times more tokens per second than one sequence on the same GPU, at nearly the same TPOT?

Remember

  • ✓Cost =((Nin−Nc)pin+Ncαpin+Noutpout)/106=\big((N_\text{in}-N_\text{c})p_\text{in}+N_\text{c}\alpha p_\text{in}+N_\text{out}p_\text{out}\big)/10^6; latency == TTFT +(Nout−1)⋅+(N_\text{out}-1)\cdotTPOT.
  • ✓Output tokens are the dearest and the slowest; cutting them is usually the first and biggest win.
  • ✓Static content first makes the prefix cacheable: cheaper input and a shorter prefill.
  • ✓Routers pick a model up front; cascades try small first and pay twice on escalation, which hurts tail latency.
  • ✓Decode is memory-bandwidth-bound at small batch: batching and quantization raise throughput almost for free.
  • ✓Optimise against your own quality bar and report latency percentiles, not means.
← back to Generative AI roadmap