Text generation

A language model factorises text as Π p(xₜ | x<ₜ) and a decoding rule picks the text. Greedy and beam search loop on likely phrases; top-k and nucleus sampling, penalties and guidance trade likelihood for diversity and control.

9 min read859 wordscore0 viewsupdated 2026-10-15

Definition

Text generation is the task of producing a token sequence x1:Tx_{1:T}, either open-ended (story, dialogue, continuation) or conditioned on an input (prompt, attribute). A neural language model gives the conditional distribution of each next token. A decoding algorithm chooses tokens from those distributions one at a time, and that choice shapes the output as much as the model does. The standard reference for open-ended decoding is Holtzman, Buys, Du, Forbes and Choi, "The Curious Case of Neural Text Degeneration" (ICLR 2020), which introduced nucleus (top-p) sampling.

Translation and summarization have one main acceptable output, so the goal is to find a high-probability sequence. Open-ended generation has many good outputs. There the goal is text that reads like human writing: fluent, varied and on topic. Before neural models, generation used templates and grammars for structured data-to-text, or sampling from smoothed nn-gram models, which were locally fluent but drifted after a few words.

The model: one factorisation, one loss

pθ(x1:T)=∏t=1Tpθ(xt∣x<t),L(θ)=−1T∑t=1Tlog⁡pθ(xt∣x<t),PPL=eLp_\theta(x_{1:T})=\prod_{t=1}^{T}p_\theta\big(x_t\mid x_{<t}\big),\qquad \mathcal L(\theta)=-\frac1T\sum_{t=1}^{T}\log p_\theta\big(x_t\mid x_{<t}\big),\qquad \mathrm{PPL}=e^{\mathcal L}
pθ(xt∣x<t)p_\theta(x_t\mid x_{<t})
softmax of the logits zt∈R∣V∣z_t\in\mathbb R^{|V|} that a causal Transformer outputs at position tt (see Language modeling and Decoder-only models)
L\mathcal L
mean next-token cross-entropy, trained with teacher forcing on real text
PPL\mathrm{PPL}
perplexity: the effective number of equally likely choices per token; lower means the model predicts held-out text better

Training never decodes. At inference the model's own choices are fed back as x<tx_{<t}, one forward pass per token (with a KV cache). Every decoding method is a rule for picking xtx_t from the processed distribution at each step.

autoregressive decoding: p(x) = Πₜ p(xₜ | x₁ … xₜ₋₁), one token per stepthecatsatonthedogcontext x₁ … xₜ₋₁ (prompt + tokens so far)decoder-only LM, causal mask → logits z ∈ ℝ^|V| for position t (toy: |V| = 12)1 · model logits zp = softmax(z)cat.398dog.239mat.139park.139others.0842 · freq. penaltyz − 0.6·count; cat: 1cat.277dog.303mat.177park.177others.0663 · T = 1, top-p 0.9keep 4, mass .934cat.297dog.325mat.189park.189others04 · sample u = 0.45cumulative intervalscat0–.297dog.297–.622 ←mat.622–.811park.811–1others—append “dog”, repeat until <eos> or the token limitstage 4 alternatives: greedy = argmax · beam = keep the B best prefixes · top-k = keep k tokenswithout stage 2, greedy picks cat (.398) and loops “the cat sat on the cat sat on …”order in Hugging Face generate: penalties → temperature → top-k → top-p → sample
fig.One step of the decoding loop with the toy model used below. After "the cat sat on the" the model gives cat 0.398 and dog 0.239. A frequency penalty of 0.6 per previous occurrence lowers cat's logit by 0.6, which flips the order to dog 0.303 and cat 0.277. Top-p 0.9 keeps four tokens holding 0.934 of the mass and renormalises. A uniform draw u = 0.45 lands in dog's interval (0.297–0.622), so dog is appended and the loop runs again.

Decoding strategies

RuleWhat it keepsDeterministicTypical use
Greedythe argmax tokenyesshort, closed tasks; fast baselines
Beam search (B beams)the B highest-scoring prefixes (see Beam search)yestranslation, summarization, ASR
Pure samplingthe whole distributionnorarely used alone: the long tail adds noise
Temperature TTreshapes: softmax(z/T)\mathrm{softmax}(z/T)noT<1T<1 sharpens, T>1T>1 flattens (see Temperature, Top-k & Top-p)
Top-kk (Fan, Lewis and Dauphin, 2018)the kk most probable tokens; they used k=10k=10nostory generation; fixed-size candidate set
Top-pp / nucleus (Holtzman et al., 2020)smallest set with mass ≥p\ge pnothe default for open-ended LLM sampling
Choosing the next token from p(· | x<t)
V(p)=arg⁡min⁡V′⊆V∣V′∣  s.t.∑x∈V′p(x∣x<t)≥p,p′(x∣x<t)=p(x∣x<t) 1[x∈V(p)]∑x′∈V(p)p(x′∣x<t)V^{(p)}=\arg\min_{V'\subseteq V}|V'|\ \ \text{s.t.}\sum_{x\in V'}p(x\mid x_{<t})\ge p,\qquad p'(x\mid x_{<t})=\frac{p(x\mid x_{<t})\,\mathbb 1[x\in V^{(p)}]}{\sum_{x'\in V^{(p)}}p(x'\mid x_{<t})}
V(p)V^{(p)}
the nucleus: the most probable tokens, added in order until their mass reaches p
∣V(p)∣|V^{(p)}|
adapts to the context: one or two tokens when the model is confident, hundreds when it is not, which a fixed k cannot do

Holtzman et al. made two observations. First, maximisation degenerates: greedy and beam continuations from GPT-2 fall into repetitive loops, and each repetition makes the next one more likely, a positive feedback loop. Second, human text is not the most probable text: real continuations have per-token probabilities that vary widely, while beam search output stays uniformly high. Pure sampling avoids the loops but often picks low-probability tokens from the tail, which the model estimates badly. Truncating the tail and sampling from the nucleus avoids both failures.

Later rules refine the truncation. Locally typical sampling (Meister et al., 2023) keeps tokens whose surprisal is close to the distribution's entropy. Min-pp keeps tokens with p(x)≥pmin⁡⋅max⁡x′p(x′)p(x)\ge p_{\min}\cdot\max_{x'}p(x'). Contrastive search (Su et al., 2022) adds a penalty for similarity to earlier hidden states.

Repetition penalties

A penalty edits the logits using the history before softmax. The frequency/presence penalty (as in the OpenAI API) is zj←zj−αfreq cj−αpres 1[cj>0]z_j\leftarrow z_j-\alpha_{\text{freq}}\,c_j-\alpha_{\text{pres}}\,\mathbb 1[c_j>0], where cjc_j is the count of token jj so far. CTRL (Keskar et al., 2019) instead divides the logits of already generated tokens by θ=1.2\theta=1.2. Hugging Face's repetition_penalty does the same, multiplying negative logits by θ\theta so the penalty always lowers them. no_repeat_ngram_size=n bans any token that would recreate an nn-gram already in the output.

These are heuristics applied at inference. Unlikelihood training (Welleck et al., ICLR 2020) fixes the cause in training: it adds a loss term −∑c∈Ctlog⁡(1−pθ(c∣x<t))-\sum_{c\in\mathcal C_t}\log\big(1-p_\theta(c\mid x_{<t})\big) that pushes down the probability of negative candidates Ct\mathcal C_t, such as tokens already in the context.

Controllable generation

p(xt∣x<t,a)  ∝  p(xt∣x<t)  p(a∣x≤t)p(x_t\mid x_{<t},a)\;\propto\;p(x_t\mid x_{<t})\;p(a\mid x_{\le t})
aa
desired attribute: topic, sentiment, formality, non-toxicity
p(a∣x≤t)p(a\mid x_{\le t})
a classifier that judges a prefix; FUDGE (Yang and Klein, 2021) trains it on prefixes so it predicts the attribute of the finished text

This is Bayes' rule applied one token at a time: rescore the top candidates by how likely each one makes the attribute. Other ways to control the output: CTRL (1.63B parameters) prepends a control code such as a domain or URL during training. PPLM (Dathathri et al., ICLR 2020) follows the gradient of an attribute classifier into the LM's hidden states. Today most control comes through prompting, Instruction tuning and RLHF, which train the model to follow a description of the desired output.

Evaluating generated text

What each measure sees

Perplexity
how well the model predicts held-out human text. It measures the model, not a decoding rule. Scoring generated text by its own model's perplexity rewards repetitive output.
Distinct-n / rep-n
share of unique n-grams, or of n-grams already seen earlier in the text. Cheap and catches loops, but blind to fluency and coherence.
MAUVE
Pillutla et al. (NeurIPS 2021): embed human and model texts, quantise them with kk-means, and measure the area under the curve of (e−c KL(Q∥Rλ),e−c KL(P∥Rλ))\big(e^{-c\,\mathrm{KL}(Q\Vert R_\lambda)},e^{-c\,\mathrm{KL}(P\Vert R_\lambda)}\big) for the mixtures Rλ=λP+(1−λ)QR_\lambda=\lambda P+(1-\lambda)Q. It is near 1 when the two distributions match and sensitive to both blandness and noise.
Human / LLM judges
pairwise preference for fluency, coherence, interest and faithfulness. This remains the reference standard; LLM judges are cheaper but biased (see LLM evaluation and benchmarking).
sampling · prompt “the” + 20 tokensT = 1.0 · top-k off · top-p 0.90 · seed 0thedogsatonthecatandthedogranandthenthecatsatinthecatsatintherose = completes a trigram seen earlier · click a token to inspect its stepstep 1: p(· | “the”) → chose “dog” · 4 of 12 tokens kept00.51thecatdogsatranoninmatparkandthen.dashed: model p · filled: after penalty, T and truncation · ✕ banned by the 3-gram rulemeasured with the raw modelPPLdist-2rep-3this run2.820.70016%100 seeds3.180.70913%human2.600.81010%human: “the dog ran in the park . …”PPL = exp(−(1/20) Σ log p(xₜ | xₜ₋₁)) under the unmodified modeldist-2 = unique bigrams / bigrams · rep-3 = share of trigrams already seen earlier in the textperplexity and diversity in the same range as the human sentencethe toy model is a bigram table over 12 words; unlisted transitions get p ≈ 0.001
decoder
fig.A 12-word bigram model generates 20 tokens after "the". Metrics always use the unmodified model. Default nucleus sampling (T = 1, top-p 0.9) averages perplexity 3.18, distinct-2 0.709 and 13% repeated trigrams over 100 seeds. A human-written sentence scores 2.60, 0.810 and 10%. Greedy and beam search (B = 4) both produce "the cat sat on the cat sat on …", with perplexity 1.83 (more likely than the human text), distinct-2 0.200 and 79% repeated trigrams. Turning on a frequency penalty of 0.6 lets greedy escape the loop (2.52, 0.750, 16%). Pure sampling (top-p off) averages perplexity 3.82 because it samples from the tail.

Worked example: one processed step

The context is "the cat sat on the", so the model is predicting the token after "the". Twelve tokens; the four shown hold 0.916 of the mass and eight others share 0.084.

TokenModel pT = 0.7Top-p 0.9Freq. penalty α = 0.6
cat0.3980.5040.4350.277
dog0.2390.2430.2610.303
mat0.1390.1130.1520.177
park0.1390.1130.1520.177
8 others0.0840.0280 (cut)0.066
Next-token probabilities under each processing step (computed in Python, matching the figure)
  1. 1T=0.7T=0.7 divides the logits (log⁡0.398=−0.92\log0.398=-0.92, log⁡0.239=−1.43\log0.239=-1.43) by 0.7 before softmax. That raises cat from 0.398 to 0.504 and shrinks the tail from 0.084 to 0.028.
  2. 2Top-p 0.9: sorted cumulative mass is 0.398, 0.637, 0.776, 0.916. The fourth token crosses 0.9, so four survive and are divided by 0.916.
  3. 3Frequency penalty: "cat" occurred once and "the" twice, so their logits drop by 0.6 and 1.2. Cat falls to −1.52-1.52, below dog's −1.43-1.43, and greedy now picks dog. Without the penalty greedy picks cat again and loops.
python
1import torch
2from transformers import AutoModelForCausalLM, AutoTokenizer
3
4tok = AutoTokenizer.from_pretrained("gpt2-large")
5lm = AutoModelForCausalLM.from_pretrained("gpt2-large").eval()
6ids = tok("The cat sat on the", return_tensors="pt").input_ids
7
8rules = {
9 "greedy": dict(do_sample=False),
10 "beam": dict(do_sample=False, num_beams=4, no_repeat_ngram_size=3),
11 "nucleus": dict(do_sample=True, top_p=0.9, temperature=1.0, top_k=0),
12 "top-k": dict(do_sample=True, top_k=40, temperature=0.7),
13 "penalty": dict(do_sample=True, top_p=0.9, repetition_penalty=1.2),
14}
15for name, kw in rules.items():
16 torch.manual_seed(0)
17 out = lm.generate(ids, max_new_tokens=40, pad_token_id=tok.eos_token_id, **kw)
18 print(name, "|", tok.decode(out[0, ids.shape[1]:]))
19
20@torch.no_grad()
21def perplexity(text): # exp of mean next-token cross-entropy
22 x = tok(text, return_tensors="pt").input_ids
23 return torch.exp(lm(x, labels=x).loss).item() # labels are shifted inside the model
Comparing decoding rules with Hugging Face generate, and scoring a text's perplexity (not run here)

Check yourself

Why does nucleus sampling adapt better than top-k with a fixed k?

Check yourself

In the worked example, the logit of 'cat' is −0.92 and it has appeared once. With a frequency penalty α = 0.6, what is its new logit, and does greedy still pick it over 'dog' (−1.43)?

Remember

  • ✓Generation = model ∏tp(xt∣x<t)\prod_t p(x_t\mid x_{<t}) + decoding rule. Training uses teacher forcing; inference feeds back the model's own tokens.
  • ✓Greedy and beam search maximise likelihood and degenerate into loops in open-ended text. Human text is not the most probable text.
  • ✓Top-kk keeps kk tokens; nucleus keeps the smallest set with mass ≥p\ge p, which adapts to the model's confidence. Temperature rescales logits first.
  • ✓Penalties edit logits from the history (zj−αcjz_j-\alpha c_j, CTRL's ÷1.2\div1.2, no-repeat nn-grams); unlikelihood training addresses repetition during training.
  • ✓Control: p(xt∣x<t,a)∝p(xt∣x<t) p(a∣x≤t)p(x_t\mid x_{<t},a)\propto p(x_t\mid x_{<t})\,p(a\mid x_{\le t}) (FUDGE), control codes (CTRL), or instructions.
  • ✓Evaluate with perplexity (the model), distinct-n/rep-n (loops), MAUVE (distribution match) and human judgments.
← back to Deep Learning roadmap