# Model families and model selection

> Pick the family by input/output contract (decoder-only, encoder, encoder–decoder, diffusion, multimodal), then the model by your own eval set: the cheapest that clears the quality bar within latency, context and licence limits.

Source: https://www.mlfoundry.in/learn/model-families-and-model-selection  
Section: Generative AI › Foundations of Generative AI  
Level: core · 8 min read · updated 2026-10-21

**Model selection:** is choosing, for one task, the model family and then the specific model that meets fixed requirements on quality, latency, cost, context length, licence and data handling at the lowest total cost. A **model family** is a group of architectures that share an input–output contract and training objective: decoder-only language models, encoder-only models, encoder–decoder models, diffusion models and multimodal models.

Teams get this wrong in two directions. They default to the largest general model for everything and pay 10–100× more per request than needed, or they pick a small model from a leaderboard and find it fails on their own data. Selection is an engineering decision, made with measurements on your task. It belongs in a system design next to prompt design and retrieval, and it is revisited whenever a new model ships.

## The families

**Decoder-only** models (see **Decoder-only models** and **GPT architecture**) predict the next token with causal attention and generate by repeating that step. They are the default for open-ended text, code, chat, tool use and RAG answers. Llama, Qwen, Mistral and GPT models belong here. **Mixture-of-experts** variants keep the same contract but route each token to a few expert feed-forward blocks. Mixtral 8×7B (Jiang et al., 2024) has 46.7B parameters but uses about 12.9B per token.

**Encoder-only** models (**BERT**, **Sentence embeddings** models, cross-encoder rerankers) read the whole input with bidirectional attention and output a vector or a label. They do not generate text, but for classification, retrieval and reranking they are often as accurate as an LLM at a tiny fraction of the cost. **Encoder–decoder** models (**T5**, **BART**, Whisper) encode an input sequence and decode a different one, a good fit for translation, transcription and rewriting.

**Diffusion** models (see **Diffusion models** and **Latent diffusion models**) generate images, video or audio by iteratively denoising a latent conditioned on a prompt. **Multimodal** models connect a vision or audio encoder to a language model (see **Vision-language models** and **Multimodal LLMs**), or align two encoders in a shared space as **CLIP** does.

> Figure: Left: five families by their input–output contract, with example models and typical jobs. The contract decides the family: if the output is a label or a vector, an encoder is usually enough; if it is free text, you need a decoder. Right: the selection flow. Hard limits (context, licence, data residency) filter before any quality test, and the final choice is the cheapest model that passes your own eval set.

Families at a glance

| Family | Objective | Output | Typical job | Article |
| --- | --- | --- | --- | --- |
| Decoder-only | next-token likelihood | text, code, tool calls | chat, generation, agents | **GPT architecture** |
| Encoder-only | masked-token prediction, contrastive | vector or label | search, classification, reranking | **BERT** |
| Encoder–decoder | sequence-to-sequence likelihood | new sequence | translation, ASR, summarisation | **T5** |
| Diffusion | denoising (noise prediction) | image, video, audio | text-to-image, editing | **Diffusion models** |
| Multimodal | contrastive or next-token over mixed inputs | text or embeddings | captioning, VQA, document reading | **Vision-language models** |

## The criteria, as numbers

Quality comes from your own eval set. The other criteria can be computed before you run anything. Cost per request for a token-priced model:

$$
c=\frac{n_\text{in}\,p_\text{in}+n_\text{out}\,p_\text{out}}{10^6},\qquad C_\text{month}=V\,c
$$

- $n_\text{in},\,n_\text{out}$: input and output tokens per request (prompt, retrieved context, answer)
- $p_\text{in},\,p_\text{out}$: price per million tokens; output tokens usually cost several times more than input
- $V$: requests per month

Latency for a streamed answer is time to first token plus one decode step per output token. At batch size 1, each decode step must read all the weights from GPU memory, which sets a floor on the time per output token (TPOT):

$$
T\approx\text{TTFT}+n_\text{out}\cdot\text{TPOT},\qquad \text{TPOT}\;\gtrsim\;\frac{N_\text{active}\cdot b}{\text{memory bandwidth}}
$$

- $\text{TTFT}$: time to first token: queueing plus prefill of the $n_\text{in}$ prompt tokens
- $N_\text{active}$: parameters used per token (all of them for a dense model, the routed experts for MoE)
- $b$: bytes per parameter: 2 for bf16, 1 for int8, about 0.5 for 4-bit (see **Quantization**)

Example: an 8B dense model in bf16 is 16 GB of weights. On a GPU with 3.35 TB/s of memory bandwidth (H100 SXM), one decode step takes at least 16/3350 s = 4.8 ms, so at most about 209 tokens/s for one stream. A 70B model in bf16 needs 140 GB for weights alone, more than one 80 GB GPU; at 4-bit it is about 35 GB. **KV cache** memory comes on top and grows with context length and batch size.

**Stable reference points**

- **BERT-base:** 110M parameters, 512-token input limit (Devlin et al., 2019)
- **Whisper:** Encoder–decoder speech model trained on 680,000 hours of labelled audio (Radford et al., 2022)
- **Stable Diffusion:** Latent diffusion (Rombach et al., CVPR 2022): a 512 × 512 image is denoised as a 64 × 64 × 4 latent, 8× downsampling per side
- **CLIP:** Dual encoder trained contrastively on 400M image–text pairs (Radford et al., 2021)
- **Mixtral 8×7B:** 8 experts, top-2 routing per token, 46.7B total and about 12.9B active parameters (Jiang et al., 2024)
- **Weight memory:** parameters × bytes per parameter: 7B ≈ 14 GB in bf16, ≈ 3.5 GB at 4-bit, before KV cache and activations

## Selecting on your constraints

> Figure: Seven hypothetical candidates (archetypes, not real products) with illustrative prices and eval scores. Defaults: RAG QA (2,000 in, 200 out tokens), 1M requests/month, quality bar 0.80, latency budget 5 s. dec-70B fails on latency (6.30 s); api-small is picked at $420/month (0.80, 1.95 s). Require in-house data: MoE-47B/13B at $720. Raise the bar to 0.85: only api-large passes, at $9,000; relax latency to 7 s and dec-70B wins at $1,580. Switch to ticket classification at a 0.90 bar and 1 s: the 110M encoder wins at $3/month (0.93, 0.01 s).

Two patterns show up. The cost spread between candidates that pass is often 10× or more, so the quality bar should be set from what the product needs, not from the best score available. And for label or vector outputs the encoder family dominates: in the classification setting it costs $\text{\textdollar}3$ a month where the largest API model costs $\text{\textdollar}975$.

## Worked example: the default RAG case

2,000 input + 200 output tokens, 1,000,000 requests/month, quality ≥ 0.80, latency ≤ 5 s. Cost = 10⁶ × (2000·p_in + 200·p_out)/10⁶; latency = TTFT + 200·TPOT. All prices and scores illustrative

| Model | p_in / p_out ($/1M) | Quality | Latency | $/month | Verdict |
| --- | --- | --- | --- | --- | --- |
| dec-3B | 0.05 / 0.10 | 0.68 | 0.05 + 1.60 = 1.65 s | 100 + 20 = 120 | quality < bar |
| dec-8B | 0.10 / 0.20 | 0.76 | 0.08 + 2.40 = 2.48 s | 200 + 40 = 240 | quality < bar |
| MoE-47B/13B | 0.30 / 0.60 | 0.82 | 0.15 + 3.00 = 3.15 s | 600 + 120 = 720 | passes |
| dec-70B | 0.70 / 0.90 | 0.86 | 0.30 + 6.00 = 6.30 s | 1,400 + 180 = 1,580 | too slow |
| api-small | 0.15 / 0.60 | 0.80 | 0.35 + 1.60 = 1.95 s | 300 + 120 = 420 | **pick** |
| api-large | 3.00 / 15.00 | 0.91 | 0.60 + 4.00 = 4.60 s | 6,000 + 3,000 = 9,000 | passes |

The pick costs 21× less than api-large, and its quality just meets the bar. On a bar that tight, check the confidence interval of the eval score: with 300 items, a score of 0.80 has a 95% interval of roughly ±0.045.

```python
cands = {
    # name: (hosted, ttft_s, tpot_s, $/1M in, $/1M out, eval quality)
    "dec-3B":      ("self", 0.05, 0.008, 0.05, 0.10, 0.68),
    "dec-8B":      ("self", 0.08, 0.012, 0.10, 0.20, 0.76),
    "MoE-47B/13B": ("self", 0.15, 0.015, 0.30, 0.60, 0.82),
    "dec-70B":     ("self", 0.30, 0.030, 0.70, 0.90, 0.86),
    "api-small":   ("api",  0.35, 0.008, 0.15, 0.60, 0.80),
    "api-large":   ("api",  0.60, 0.020, 3.00, 15.0, 0.91),
}

def choose(cands, n_in, n_out, volume, q_bar, lat_budget, in_house=False):
    passing = []
    for name, (hosted, ttft, tpot, p_in, p_out, q) in cands.items():
        latency = ttft + n_out * tpot
        monthly = volume * (n_in * p_in + n_out * p_out) / 1e6
        if q >= q_bar and latency <= lat_budget and not (in_house and hosted == "api"):
            passing.append((monthly, name))
    return min(passing)[1] if passing else None

print(choose(cands, 2000, 200, 1_000_000, q_bar=0.80, lat_budget=5.0))                 # api-small
print(choose(cands, 2000, 200, 1_000_000, q_bar=0.80, lat_budget=5.0, in_house=True))  # MoE-47B/13B
```

Filter candidates on hard limits and the quality bar, then take the cheapest; eval scores come from running each model on your own labelled set (runs as is)

> **Leaderboards are not your eval:** Public rankings change every few months, mix tasks you do not care about, and can be inflated by test data leaking into training sets. Use them only to build a shortlist. Decide on a few hundred labelled examples from your own traffic, scored the same way for every candidate (see **LLM evaluation and benchmarking**).

> **One model is rarely the answer:** Many systems route: an encoder classifies intent, a small model handles routine requests, and only hard or low-confidence cases go to a large model. If 80% of traffic can stop at a model costing 1/20 as much, the blended cost falls to 0.8 × 0.05 + 0.2 × 1 = 0.24 of always using the large one.

**Strengths**

- Smaller or specialised models: lower cost and latency, can run on your own hardware, easier to fine-tune.
- Larger general models: higher quality on open-ended and multi-step tasks, fewer prompt workarounds.

**Limitations**

- Smaller models fail more often on reasoning, long context and rare knowledge; check on your data.
- Larger models cost more per token, are slower per stream, and may only be available through an API (see **Open-weight vs closed models**).

**Question:** You need to route 5M support tickets a month into 40 categories, and you have 20,000 labelled tickets. Which family should you try first?

- A. A large decoder-only LLM with few-shot prompts
- B. A fine-tuned encoder-only classifier
- C. A diffusion model
- D. An encoder–decoder translation model

**Answer:** B. A fine-tuned encoder-only classifier — The output is a label and labelled data exists. A fine-tuned encoder reads 300 tokens in milliseconds for a tiny fraction of an LLM's cost. Try the LLM only if the encoder misses the quality bar.

**Question:** A dense 13B model in bf16 runs at batch size 1 on a GPU with 2 TB/s of memory bandwidth. What is the best possible decode speed for one stream?

- A. About 15 tokens/s
- B. About 77 tokens/s
- C. About 154 tokens/s
- D. About 2,000 tokens/s

**Answer:** B. About 77 tokens/s — Weights are $13\times10^9\times2=26$ GB. Each token reads them once: $26/2000=13$ ms, so at most about 77 tokens/s. Batching raises total throughput, but not the speed of one stream.

**Key points**

- Choose the family by input–output contract: decoder for free text, encoder for labels and vectors, encoder–decoder for sequence-to-sequence, diffusion for media, multimodal for mixed inputs.
- Cost per request $=(n_\text{in}p_\text{in}+n_\text{out}p_\text{out})/10^6$; latency $\approx$ TTFT $+\,n_\text{out}\cdot$TPOT; TPOT at batch 1 is bounded by active weight bytes ÷ memory bandwidth.
- Apply hard limits first (context, licence, data residency), then pick the cheapest model that clears the quality bar on your own eval set.
- Costs among passing models often differ by 10× or more; routing easy traffic to small models cuts the blended cost.
- Rankings and prices change fast: re-run the eval when candidates change, rather than trusting leaderboards.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
