Definition
Text generation is the task of producing a token sequence , either open-ended (story, dialogue, continuation) or conditioned on an input (prompt, attribute). A neural language model gives the conditional distribution of each next token. A decoding algorithm chooses tokens from those distributions one at a time, and that choice shapes the output as much as the model does. The standard reference for open-ended decoding is Holtzman, Buys, Du, Forbes and Choi, "The Curious Case of Neural Text Degeneration" (ICLR 2020), which introduced nucleus (top-p) sampling.
Translation and summarization have one main acceptable output, so the goal is to find a high-probability sequence. Open-ended generation has many good outputs. There the goal is text that reads like human writing: fluent, varied and on topic. Before neural models, generation used templates and grammars for structured data-to-text, or sampling from smoothed -gram models, which were locally fluent but drifted after a few words.
The model: one factorisation, one loss
- softmax of the logits that a causal Transformer outputs at position (see Language modeling and Decoder-only models)
- mean next-token cross-entropy, trained with teacher forcing on real text
- perplexity: the effective number of equally likely choices per token; lower means the model predicts held-out text better
Training never decodes. At inference the model's own choices are fed back as , one forward pass per token (with a KV cache). Every decoding method is a rule for picking from the processed distribution at each step.
Decoding strategies
| Rule | What it keeps | Deterministic | Typical use |
|---|---|---|---|
| Greedy | the argmax token | yes | short, closed tasks; fast baselines |
| Beam search (B beams) | the B highest-scoring prefixes (see Beam search) | yes | translation, summarization, ASR |
| Pure sampling | the whole distribution | no | rarely used alone: the long tail adds noise |
| Temperature | reshapes: | no | sharpens, flattens (see Temperature, Top-k & Top-p) |
| Top- (Fan, Lewis and Dauphin, 2018) | the most probable tokens; they used | no | story generation; fixed-size candidate set |
| Top- / nucleus (Holtzman et al., 2020) | smallest set with mass | no | the default for open-ended LLM sampling |
- the nucleus: the most probable tokens, added in order until their mass reaches p
- adapts to the context: one or two tokens when the model is confident, hundreds when it is not, which a fixed k cannot do
Holtzman et al. made two observations. First, maximisation degenerates: greedy and beam continuations from GPT-2 fall into repetitive loops, and each repetition makes the next one more likely, a positive feedback loop. Second, human text is not the most probable text: real continuations have per-token probabilities that vary widely, while beam search output stays uniformly high. Pure sampling avoids the loops but often picks low-probability tokens from the tail, which the model estimates badly. Truncating the tail and sampling from the nucleus avoids both failures.
Later rules refine the truncation. Locally typical sampling (Meister et al., 2023) keeps tokens whose surprisal is close to the distribution's entropy. Min- keeps tokens with . Contrastive search (Su et al., 2022) adds a penalty for similarity to earlier hidden states.
Repetition penalties
A penalty edits the logits using the history before softmax. The frequency/presence penalty (as in the OpenAI API) is , where is the count of token so far. CTRL (Keskar et al., 2019) instead divides the logits of already generated tokens by . Hugging Face's repetition_penalty does the same, multiplying negative logits by so the penalty always lowers them. no_repeat_ngram_size=n bans any token that would recreate an -gram already in the output.
These are heuristics applied at inference. Unlikelihood training (Welleck et al., ICLR 2020) fixes the cause in training: it adds a loss term that pushes down the probability of negative candidates , such as tokens already in the context.
Controllable generation
- desired attribute: topic, sentiment, formality, non-toxicity
- a classifier that judges a prefix; FUDGE (Yang and Klein, 2021) trains it on prefixes so it predicts the attribute of the finished text
This is Bayes' rule applied one token at a time: rescore the top candidates by how likely each one makes the attribute. Other ways to control the output: CTRL (1.63B parameters) prepends a control code such as a domain or URL during training. PPLM (Dathathri et al., ICLR 2020) follows the gradient of an attribute classifier into the LM's hidden states. Today most control comes through prompting, Instruction tuning and RLHF, which train the model to follow a description of the desired output.
Evaluating generated text
What each measure sees
- Perplexity
- how well the model predicts held-out human text. It measures the model, not a decoding rule. Scoring generated text by its own model's perplexity rewards repetitive output.
- Distinct-n / rep-n
- share of unique n-grams, or of n-grams already seen earlier in the text. Cheap and catches loops, but blind to fluency and coherence.
- MAUVE
- Pillutla et al. (NeurIPS 2021): embed human and model texts, quantise them with -means, and measure the area under the curve of for the mixtures . It is near 1 when the two distributions match and sensitive to both blandness and noise.
- Human / LLM judges
- pairwise preference for fluency, coherence, interest and faithfulness. This remains the reference standard; LLM judges are cheaper but biased (see LLM evaluation and benchmarking).
Worked example: one processed step
The context is "the cat sat on the", so the model is predicting the token after "the". Twelve tokens; the four shown hold 0.916 of the mass and eight others share 0.084.
| Token | Model p | T = 0.7 | Top-p 0.9 | Freq. penalty α = 0.6 |
|---|---|---|---|---|
| cat | 0.398 | 0.504 | 0.435 | 0.277 |
| dog | 0.239 | 0.243 | 0.261 | 0.303 |
| mat | 0.139 | 0.113 | 0.152 | 0.177 |
| park | 0.139 | 0.113 | 0.152 | 0.177 |
| 8 others | 0.084 | 0.028 | 0 (cut) | 0.066 |
- 1 divides the logits (, ) by 0.7 before softmax. That raises cat from 0.398 to 0.504 and shrinks the tail from 0.084 to 0.028.
- 2Top-p 0.9: sorted cumulative mass is 0.398, 0.637, 0.776, 0.916. The fourth token crosses 0.9, so four survive and are divided by 0.916.
- 3Frequency penalty: "cat" occurred once and "the" twice, so their logits drop by 0.6 and 1.2. Cat falls to , below dog's , and greedy now picks dog. Without the penalty greedy picks cat again and loops.
1import torch2from transformers import AutoModelForCausalLM, AutoTokenizer34tok = AutoTokenizer.from_pretrained("gpt2-large")5lm = AutoModelForCausalLM.from_pretrained("gpt2-large").eval()6ids = tok("The cat sat on the", return_tensors="pt").input_ids78rules = {9 "greedy": dict(do_sample=False),10 "beam": dict(do_sample=False, num_beams=4, no_repeat_ngram_size=3),11 "nucleus": dict(do_sample=True, top_p=0.9, temperature=1.0, top_k=0),12 "top-k": dict(do_sample=True, top_k=40, temperature=0.7),13 "penalty": dict(do_sample=True, top_p=0.9, repetition_penalty=1.2),14}15for name, kw in rules.items():16 torch.manual_seed(0)17 out = lm.generate(ids, max_new_tokens=40, pad_token_id=tok.eos_token_id, **kw)18 print(name, "|", tok.decode(out[0, ids.shape[1]:]))1920@torch.no_grad()21def perplexity(text): # exp of mean next-token cross-entropy22 x = tok(text, return_tensors="pt").input_ids23 return torch.exp(lm(x, labels=x).loss).item() # labels are shifted inside the modelCheck yourself
Why does nucleus sampling adapt better than top-k with a fixed k?
Check yourself
In the worked example, the logit of 'cat' is −0.92 and it has appeared once. With a frequency penalty α = 0.6, what is its new logit, and does greedy still pick it over 'dog' (−1.43)?
Remember
- ✓Generation = model + decoding rule. Training uses teacher forcing; inference feeds back the model's own tokens.
- ✓Greedy and beam search maximise likelihood and degenerate into loops in open-ended text. Human text is not the most probable text.
- ✓Top- keeps tokens; nucleus keeps the smallest set with mass , which adapts to the model's confidence. Temperature rescales logits first.
- ✓Penalties edit logits from the history (, CTRL's , no-repeat -grams); unlikelihood training addresses repetition during training.
- ✓Control: (FUDGE), control codes (CTRL), or instructions.
- ✓Evaluate with perplexity (the model), distinct-n/rep-n (loops), MAUVE (distribution match) and human judgments.