Definition
Model selection is choosing, for one task, the model family and then the specific model that meets fixed requirements on quality, latency, cost, context length, licence and data handling at the lowest total cost. A model family is a group of architectures that share an input–output contract and training objective: decoder-only language models, encoder-only models, encoder–decoder models, diffusion models and multimodal models.
Teams get this wrong in two directions. They default to the largest general model for everything and pay 10–100× more per request than needed, or they pick a small model from a leaderboard and find it fails on their own data. Selection is an engineering decision, made with measurements on your task. It belongs in a system design next to prompt design and retrieval, and it is revisited whenever a new model ships.
The families
Decoder-only models (see Decoder-only models and GPT architecture) predict the next token with causal attention and generate by repeating that step. They are the default for open-ended text, code, chat, tool use and RAG answers. Llama, Qwen, Mistral and GPT models belong here. Mixture-of-experts variants keep the same contract but route each token to a few expert feed-forward blocks. Mixtral 8×7B (Jiang et al., 2024) has 46.7B parameters but uses about 12.9B per token.
Encoder-only models (BERT, Sentence embeddings models, cross-encoder rerankers) read the whole input with bidirectional attention and output a vector or a label. They do not generate text, but for classification, retrieval and reranking they are often as accurate as an LLM at a tiny fraction of the cost. Encoder–decoder models (T5, BART, Whisper) encode an input sequence and decode a different one, a good fit for translation, transcription and rewriting.
Diffusion models (see Diffusion models and Latent diffusion models) generate images, video or audio by iteratively denoising a latent conditioned on a prompt. Multimodal models connect a vision or audio encoder to a language model (see Vision-language models and Multimodal LLMs), or align two encoders in a shared space as CLIP does.
| Family | Objective | Output | Typical job | Article |
|---|---|---|---|---|
| Decoder-only | next-token likelihood | text, code, tool calls | chat, generation, agents | GPT architecture |
| Encoder-only | masked-token prediction, contrastive | vector or label | search, classification, reranking | BERT |
| Encoder–decoder | sequence-to-sequence likelihood | new sequence | translation, ASR, summarisation | T5 |
| Diffusion | denoising (noise prediction) | image, video, audio | text-to-image, editing | Diffusion models |
| Multimodal | contrastive or next-token over mixed inputs | text or embeddings | captioning, VQA, document reading | Vision-language models |
The criteria, as numbers
Quality comes from your own eval set. The other criteria can be computed before you run anything. Cost per request for a token-priced model:
- input and output tokens per request (prompt, retrieved context, answer)
- price per million tokens; output tokens usually cost several times more than input
- requests per month
Latency for a streamed answer is time to first token plus one decode step per output token. At batch size 1, each decode step must read all the weights from GPU memory, which sets a floor on the time per output token (TPOT):
- time to first token: queueing plus prefill of the prompt tokens
- parameters used per token (all of them for a dense model, the routed experts for MoE)
- bytes per parameter: 2 for bf16, 1 for int8, about 0.5 for 4-bit (see Quantization)
Example: an 8B dense model in bf16 is 16 GB of weights. On a GPU with 3.35 TB/s of memory bandwidth (H100 SXM), one decode step takes at least 16/3350 s = 4.8 ms, so at most about 209 tokens/s for one stream. A 70B model in bf16 needs 140 GB for weights alone, more than one 80 GB GPU; at 4-bit it is about 35 GB. KV cache memory comes on top and grows with context length and batch size.
Stable reference points
- BERT-base
- 110M parameters, 512-token input limit (Devlin et al., 2019)
- Whisper
- Encoder–decoder speech model trained on 680,000 hours of labelled audio (Radford et al., 2022)
- Stable Diffusion
- Latent diffusion (Rombach et al., CVPR 2022): a 512 × 512 image is denoised as a 64 × 64 × 4 latent, 8× downsampling per side
- CLIP
- Dual encoder trained contrastively on 400M image–text pairs (Radford et al., 2021)
- Mixtral 8×7B
- 8 experts, top-2 routing per token, 46.7B total and about 12.9B active parameters (Jiang et al., 2024)
- Weight memory
- parameters × bytes per parameter: 7B ≈ 14 GB in bf16, ≈ 3.5 GB at 4-bit, before KV cache and activations
Selecting on your constraints
Two patterns show up. The cost spread between candidates that pass is often 10× or more, so the quality bar should be set from what the product needs, not from the best score available. And for label or vector outputs the encoder family dominates: in the classification setting it costs a month where the largest API model costs .
Worked example: the default RAG case
| Model | p_in / p_out ($/1M) | Quality | Latency | $/month | Verdict |
|---|---|---|---|---|---|
| dec-3B | 0.05 / 0.10 | 0.68 | 0.05 + 1.60 = 1.65 s | 100 + 20 = 120 | quality < bar |
| dec-8B | 0.10 / 0.20 | 0.76 | 0.08 + 2.40 = 2.48 s | 200 + 40 = 240 | quality < bar |
| MoE-47B/13B | 0.30 / 0.60 | 0.82 | 0.15 + 3.00 = 3.15 s | 600 + 120 = 720 | passes |
| dec-70B | 0.70 / 0.90 | 0.86 | 0.30 + 6.00 = 6.30 s | 1,400 + 180 = 1,580 | too slow |
| api-small | 0.15 / 0.60 | 0.80 | 0.35 + 1.60 = 1.95 s | 300 + 120 = 420 | pick |
| api-large | 3.00 / 15.00 | 0.91 | 0.60 + 4.00 = 4.60 s | 6,000 + 3,000 = 9,000 | passes |
The pick costs 21× less than api-large, and its quality just meets the bar. On a bar that tight, check the confidence interval of the eval score: with 300 items, a score of 0.80 has a 95% interval of roughly ±0.045.
1cands = {2 # name: (hosted, ttft_s, tpot_s, $/1M in, $/1M out, eval quality)3 "dec-3B": ("self", 0.05, 0.008, 0.05, 0.10, 0.68),4 "dec-8B": ("self", 0.08, 0.012, 0.10, 0.20, 0.76),5 "MoE-47B/13B": ("self", 0.15, 0.015, 0.30, 0.60, 0.82),6 "dec-70B": ("self", 0.30, 0.030, 0.70, 0.90, 0.86),7 "api-small": ("api", 0.35, 0.008, 0.15, 0.60, 0.80),8 "api-large": ("api", 0.60, 0.020, 3.00, 15.0, 0.91),9}1011def choose(cands, n_in, n_out, volume, q_bar, lat_budget, in_house=False):12 passing = []13 for name, (hosted, ttft, tpot, p_in, p_out, q) in cands.items():14 latency = ttft + n_out * tpot15 monthly = volume * (n_in * p_in + n_out * p_out) / 1e616 if q >= q_bar and latency <= lat_budget and not (in_house and hosted == "api"):17 passing.append((monthly, name))18 return min(passing)[1] if passing else None1920print(choose(cands, 2000, 200, 1_000_000, q_bar=0.80, lat_budget=5.0)) # api-small21print(choose(cands, 2000, 200, 1_000_000, q_bar=0.80, lat_budget=5.0, in_house=True)) # MoE-47B/13BUse when
- +Smaller or specialised models: lower cost and latency, can run on your own hardware, easier to fine-tune.
- +Larger general models: higher quality on open-ended and multi-step tasks, fewer prompt workarounds.
Avoid when
- −Smaller models fail more often on reasoning, long context and rare knowledge; check on your data.
- −Larger models cost more per token, are slower per stream, and may only be available through an API (see Open-weight vs closed models).
Check yourself
You need to route 5M support tickets a month into 40 categories, and you have 20,000 labelled tickets. Which family should you try first?
Check yourself
A dense 13B model in bf16 runs at batch size 1 on a GPU with 2 TB/s of memory bandwidth. What is the best possible decode speed for one stream?
Remember
- ✓Choose the family by input–output contract: decoder for free text, encoder for labels and vectors, encoder–decoder for sequence-to-sequence, diffusion for media, multimodal for mixed inputs.
- ✓Cost per request ; latency TTFT TPOT; TPOT at batch 1 is bounded by active weight bytes ÷ memory bandwidth.
- ✓Apply hard limits first (context, licence, data residency), then pick the cheapest model that clears the quality bar on your own eval set.
- ✓Costs among passing models often differ by 10× or more; routing easy traffic to small models cuts the blended cost.
- ✓Rankings and prices change fast: re-run the eval when candidates change, rather than trusting leaderboards.