Definition
Cost and latency optimization is the engineering of an LLM application so that each request uses fewer paid tokens and fewer GPU-seconds, and returns sooner, while quality stays above a fixed bar. The main levers are prompt length, output length, caching, model choice and routing (FrugalGPT, Chen, Zaharia and Zou, 2023), and serving-side techniques such as batching, quantization and speculative decoding.
A chat or RAG product pays for every token twice: once in money, through per-token API prices or GPU rental, and once in time, through the wait the user sees. Both scale with traffic, so a design that is fine for a demo can cost thousands of dollars a day in production. You optimise against a quality metric from your own evaluation set. Cutting cost by moving to a model that fails a third of your cases is not an optimisation.
Two different latencies matter. Time to first token (TTFT) is how long the user waits before anything appears. It is dominated by queueing and by prefill, the parallel pass over the whole prompt. Time per output token (TPOT), also called inter-token latency, is the step time of decode, which generates one token at a time. With Streaming generation the user perceives TTFT. A program that needs the full JSON perceives the total.
The two formulas
- input and output tokens of the request
- input tokens served from a prompt cache
- price multiplier for cached input, well below 1 (0.1 in this article's examples)
- price per million tokens; output is usually several times dearer than input
- prompt tokens processed per second; prefill is compute-bound
- decode step time; at small batch sizes it is memory-bandwidth-bound
The second bound explains much of serving. An 8B-parameter model in bf16 is 16 GB of weights. An H100 SXM reads about 3.35 TB/s from HBM, so one decode step cannot take less than about 4.8 ms, or about 209 tokens/s for a single sequence. A batch of 32 sequences reads the same weights once per step, so throughput rises almost 32-fold until compute or KV-cache reads take over. That is why Continuous batching and Quantization (fewer bytes per weight) pay off so well.
The levers, in order of usual payoff
- 1Measure first. Log input, cached and output tokens, TTFT and total latency per request type. Find the endpoint that dominates spend before you touch anything.
- 2Cut output tokens. Ask for terse formats, set
max_tokens, use stop sequences, and return structured fields rather than prose. Output tokens cost the most and each one adds a decode step. - 3Make prompts cacheable. Put the system prompt, tool schemas and shared documents first and the variable part last, so Prompt caching can reuse the prefix. For repeated questions, a Semantic caching hit skips the model entirely.
- 4Shrink the input. Retrieve fewer, better chunks (see Reranking), drop stale chat turns, and compress long contexts. LLMLingua (Jiang et al., 2023) drops low-information tokens and reports up to 20× compression with small quality loss.
- 5Route or cascade. Send easy requests to a small model. A router predicts difficulty up front. A cascade tries the small model and escalates when a scorer is not confident (see Model families and model selection).
- 6Move work off the critical path. Non-interactive jobs (evaluation, back-fills, nightly summaries) go to a batch endpoint at a discount.
- 7If you self-host, tune the server. Use continuous batching and PagedAttention (see Inference servers (vLLM, TGI)), weight and KV-cache quantization, Speculative decoding for latency, and Knowledge distillation into a smaller model for a narrow task.
Checkable specifics (as of late 2024; prices and policies change)
- Output vs input price
- GPT-4o listed USD 2.50 / 10 and Claude 3.5 Sonnet USD 3 / 15 per million input/output tokens: output 4–5× dearer
- Batch APIs
- OpenAI's Batch API (April 2024) and Anthropic's Message Batches API (October 2024) both charge 50% of the normal price, with results returned asynchronously, within 24 hours
- Prompt caching
- OpenAI caches prompts of 1,024+ tokens automatically, with cached input at 50% off. Anthropic bills cache reads at 10% of the input price and cache writes at 125%
- FrugalGPT
- an LLM cascade with a learned scorer; the paper reports matching the best single model's accuracy at up to 98% lower cost on its benchmarks
- Decode bound
- 8B params × 2 bytes ÷ 3.35 TB/s ≈ 4.8 ms per step at batch 1 on one H100 SXM; a 4-bit copy reads a quarter of the bytes
Simulating routing and caching
The figure below simulates 1,000 requests with difficulty . The large model answers correctly with probability , the small one with . The router sees a noisy estimate and sends the request to the small model when . The cascade always runs the small model first, then escalates when a noisy confidence score falls below , paying for both calls.
Routing and caching act on different terms of the formulas, so their savings multiply. A cascade spends money on every escalated request and adds the small model's full latency to it. It is the right tool when the small model handles most of the traffic and its mistakes can be detected cheaply. A router decides before any generation, so it never pays twice, but it must predict difficulty from the prompt alone.
Worked example
| Step | Input cost | Output cost | Total / request | Latency |
|---|---|---|---|---|
| Baseline | 2,000 × 0.00600 | 300 × 0.00450 | $0.01050 | 0.50 + 299 × 0.025 = 7.98 s |
| + cache 1,600-token prefix | 400 × 0.30/1M = $0.00168 | $0.00450 | $0.00618 (−41%) | 0.30 + 7.48 = 7.78 s |
| + cap answer at 150 tokens | $0.00168 | 150 × 0.00225 | $0.00393 (−63%) | 0.30 + 149 × 0.025 = 4.03 s |
| + router, τ = 0.40 | mixed | mixed | $0.00242 (−77%) | mean 2.93 s, p95 4.03 s |
At one million requests a month the baseline costs 2,420, at a cost of two accuracy points on this toy distribution. Whether that trade is acceptable is a product decision. Make it on your own evaluation set, not on a benchmark.
1PRICES = { # $ per 1M tokens: (input, cached input, output); illustrative2 "large": (3.00, 0.30, 15.00),3 "small": (0.15, 0.015, 0.60),4}56def request_cost(model, usage):7 p_in, p_cached, p_out = PRICES[model]8 cached = usage.get("cached_tokens", 0)9 fresh = usage["input_tokens"] - cached10 return (fresh * p_in + cached * p_cached + usage["output_tokens"] * p_out) / 1e61112def latency(ttft_s, tpot_s, out_tokens):13 return ttft_s + (out_tokens - 1) * tpot_s1415def cascade(question, small, large, judge, tau=0.7):16 """Small model first; escalate to the large one when the judge is not confident."""17 draft, u1 = small(question)18 if judge(question, draft) >= tau:19 return draft, request_cost("small", u1)20 final, u2 = large(question)21 return final, request_cost("small", u1) + request_cost("large", u2) # paid twice2223usage = {"input_tokens": 2000, "cached_tokens": 1600, "output_tokens": 150}24print(round(request_cost("large", usage), 5)) # 0.0039325print(round(request_cost("small", usage), 6)) # 0.00017426print(round(latency(0.30, 0.025, 150), 3)) # 4.025Use when
- +Output limits, prompt order for caching and batch endpoints need no model change and keep quality intact.
- +Routing and caching compound with each other and with lower token prices.
- +Self-hosted quantization and batching cut GPU-hours by large factors on bandwidth-bound decode.
Avoid when
- −Routers and cascades add a component that can silently misroute hard requests; they need their own evaluation.
- −Aggressive prompt compression or chunk pruning can remove the one fact the answer needed.
- −Every cache adds staleness and invalidation problems; semantic caches can return a wrong answer to a similar-looking question.
Check yourself
A request has 3,000 input tokens and 600 output tokens; prefill runs at 10,000 tokens/s with 0.2 s overhead and TPOT is 20 ms. What dominates its latency?
Check yourself
Why does a batch of 16 sequences decode almost 16 times more tokens per second than one sequence on the same GPU, at nearly the same TPOT?
Remember
- ✓Cost ; latency TTFT TPOT.
- ✓Output tokens are the dearest and the slowest; cutting them is usually the first and biggest win.
- ✓Static content first makes the prefix cacheable: cheaper input and a shorter prefill.
- ✓Routers pick a model up front; cascades try small first and pay twice on escalation, which hurts tail latency.
- ✓Decode is memory-bandwidth-bound at small batch: batching and quantization raise throughput almost for free.
- ✓Optimise against your own quality bar and report latency percentiles, not means.