# Text generation

> A language model factorises text as Π p(xₜ | x<ₜ) and a decoding rule picks the text. Greedy and beam search loop on likely phrases; top-k and nucleus sampling, penalties and guidance trade likelihood for diversity and control.

Source: https://www.mlfoundry.in/learn/text-generation  
Section: Deep Learning › Deep Learning for NLP  
Level: core · 9 min read · updated 2026-10-15

**Text generation:** is the task of producing a token sequence $x_{1:T}$, either open-ended (story, dialogue, continuation) or conditioned on an input (prompt, attribute). A neural language model gives the conditional distribution of each next token. A **decoding algorithm** chooses tokens from those distributions one at a time, and that choice shapes the output as much as the model does. The standard reference for open-ended decoding is Holtzman, Buys, Du, Forbes and Choi, "The Curious Case of Neural Text Degeneration" (ICLR 2020), which introduced nucleus (top-p) sampling.

Translation and summarization have one main acceptable output, so the goal is to find a high-probability sequence. Open-ended generation has many good outputs. There the goal is text that reads like human writing: fluent, varied and on topic. Before neural models, generation used templates and grammars for structured data-to-text, or sampling from smoothed $n$-gram models, which were locally fluent but drifted after a few words.

## The model: one factorisation, one loss

$$
p_\theta(x_{1:T})=\prod_{t=1}^{T}p_\theta\big(x_t\mid x_{<t}\big),\qquad \mathcal L(\theta)=-\frac1T\sum_{t=1}^{T}\log p_\theta\big(x_t\mid x_{<t}\big),\qquad \mathrm{PPL}=e^{\mathcal L}
$$

- $p_\theta(x_t\mid x_{<t})$: softmax of the logits $z_t\in\mathbb R^{|V|}$ that a causal Transformer outputs at position $t$ (see **Language modeling** and **Decoder-only models**)
- $\mathcal L$: mean next-token cross-entropy, trained with teacher forcing on real text
- $\mathrm{PPL}$: perplexity: the effective number of equally likely choices per token; lower means the model predicts held-out text better

Training never decodes. At inference the model's own choices are fed back as $x_{<t}$, one forward pass per token (with a **KV cache**). Every decoding method is a rule for picking $x_t$ from the processed distribution at each step.

> Figure: One step of the decoding loop with the toy model used below. After "the cat sat on the" the model gives cat 0.398 and dog 0.239. A frequency penalty of 0.6 per previous occurrence lowers cat's logit by 0.6, which flips the order to dog 0.303 and cat 0.277. Top-p 0.9 keeps four tokens holding 0.934 of the mass and renormalises. A uniform draw u = 0.45 lands in dog's interval (0.297–0.622), so dog is appended and the loop runs again.

## Decoding strategies

Choosing the next token from p(· | x<t)

| Rule | What it keeps | Deterministic | Typical use |
| --- | --- | --- | --- |
| Greedy | the argmax token | yes | short, closed tasks; fast baselines |
| Beam search (B beams) | the B highest-scoring prefixes (see **Beam search**) | yes | translation, summarization, ASR |
| Pure sampling | the whole distribution | no | rarely used alone: the long tail adds noise |
| Temperature $T$ | reshapes: $\mathrm{softmax}(z/T)$ | no | $T<1$ sharpens, $T>1$ flattens (see **Temperature, Top-k & Top-p**) |
| Top-$k$ (Fan, Lewis and Dauphin, 2018) | the $k$ most probable tokens; they used $k=10$ | no | story generation; fixed-size candidate set |
| Top-$p$ / nucleus (Holtzman et al., 2020) | smallest set with mass $\ge p$ | no | the default for open-ended LLM sampling |

$$
V^{(p)}=\arg\min_{V'\subseteq V}|V'|\ \ \text{s.t.}\sum_{x\in V'}p(x\mid x_{<t})\ge p,\qquad p'(x\mid x_{<t})=\frac{p(x\mid x_{<t})\,\mathbb 1[x\in V^{(p)}]}{\sum_{x'\in V^{(p)}}p(x'\mid x_{<t})}
$$

- $V^{(p)}$: the nucleus: the most probable tokens, added in order until their mass reaches p
- $|V^{(p)}|$: adapts to the context: one or two tokens when the model is confident, hundreds when it is not, which a fixed k cannot do

Holtzman et al. made two observations. First, **maximisation degenerates**: greedy and beam continuations from GPT-2 fall into repetitive loops, and each repetition makes the next one more likely, a positive feedback loop. Second, **human text is not the most probable text**: real continuations have per-token probabilities that vary widely, while beam search output stays uniformly high. Pure sampling avoids the loops but often picks low-probability tokens from the tail, which the model estimates badly. Truncating the tail and sampling from the nucleus avoids both failures.

Later rules refine the truncation. Locally typical sampling (Meister et al., 2023) keeps tokens whose surprisal is close to the distribution's entropy. Min-$p$ keeps tokens with $p(x)\ge p_{\min}\cdot\max_{x'}p(x')$. Contrastive search (Su et al., 2022) adds a penalty for similarity to earlier hidden states.

## Repetition penalties

A penalty edits the logits using the history before softmax. The **frequency/presence penalty** (as in the OpenAI API) is $z_j\leftarrow z_j-\alpha_{\text{freq}}\,c_j-\alpha_{\text{pres}}\,\mathbb 1[c_j>0]$, where $c_j$ is the count of token $j$ so far. CTRL (Keskar et al., 2019) instead divides the logits of already generated tokens by $\theta=1.2$. Hugging Face's `repetition_penalty` does the same, multiplying negative logits by $\theta$ so the penalty always lowers them. `no_repeat_ngram_size=n` bans any token that would recreate an $n$-gram already in the output.

These are heuristics applied at inference. **Unlikelihood training** (Welleck et al., ICLR 2020) fixes the cause in training: it adds a loss term $-\sum_{c\in\mathcal C_t}\log\big(1-p_\theta(c\mid x_{<t})\big)$ that pushes down the probability of negative candidates $\mathcal C_t$, such as tokens already in the context.

## Controllable generation

$$
p(x_t\mid x_{<t},a)\;\propto\;p(x_t\mid x_{<t})\;p(a\mid x_{\le t})
$$

- $a$: desired attribute: topic, sentiment, formality, non-toxicity
- $p(a\mid x_{\le t})$: a classifier that judges a prefix; FUDGE (Yang and Klein, 2021) trains it on prefixes so it predicts the attribute of the finished text

This is Bayes' rule applied one token at a time: rescore the top candidates by how likely each one makes the attribute. Other ways to control the output: CTRL (1.63B parameters) prepends a **control code** such as a domain or URL during training. PPLM (Dathathri et al., ICLR 2020) follows the gradient of an attribute classifier into the LM's hidden states. Today most control comes through prompting, **Instruction tuning** and **RLHF**, which train the model to follow a description of the desired output.

## Evaluating generated text

**What each measure sees**

- **Perplexity:** how well the model predicts held-out human text. It measures the model, not a decoding rule. Scoring generated text by its own model's perplexity rewards repetitive output.
- **Distinct-n / rep-n:** share of unique n-grams, or of n-grams already seen earlier in the text. Cheap and catches loops, but blind to fluency and coherence.
- **MAUVE:** Pillutla et al. (NeurIPS 2021): embed human and model texts, quantise them with $k$-means, and measure the area under the curve of $\big(e^{-c\,\mathrm{KL}(Q\Vert R_\lambda)},e^{-c\,\mathrm{KL}(P\Vert R_\lambda)}\big)$ for the mixtures $R_\lambda=\lambda P+(1-\lambda)Q$. It is near 1 when the two distributions match and sensitive to both blandness and noise.
- **Human / LLM judges:** pairwise preference for fluency, coherence, interest and faithfulness. This remains the reference standard; LLM judges are cheaper but biased (see **LLM evaluation and benchmarking**).

> Figure: A 12-word bigram model generates 20 tokens after "the". Metrics always use the unmodified model. Default nucleus sampling (T = 1, top-p 0.9) averages perplexity 3.18, distinct-2 0.709 and 13% repeated trigrams over 100 seeds. A human-written sentence scores 2.60, 0.810 and 10%. Greedy and beam search (B = 4) both produce "the cat sat on the cat sat on …", with perplexity 1.83 (more likely than the human text), distinct-2 0.200 and 79% repeated trigrams. Turning on a frequency penalty of 0.6 lets greedy escape the loop (2.52, 0.750, 16%). Pure sampling (top-p off) averages perplexity 3.82 because it samples from the tail.

## Worked example: one processed step

The context is "the cat sat on the", so the model is predicting the token after "the". Twelve tokens; the four shown hold 0.916 of the mass and eight others share 0.084.

Next-token probabilities under each processing step (computed in Python, matching the figure)

| Token | Model p | T = 0.7 | Top-p 0.9 | Freq. penalty α = 0.6 |
| --- | --- | --- | --- | --- |
| cat | 0.398 | 0.504 | 0.435 | 0.277 |
| dog | 0.239 | 0.243 | 0.261 | 0.303 |
| mat | 0.139 | 0.113 | 0.152 | 0.177 |
| park | 0.139 | 0.113 | 0.152 | 0.177 |
| 8 others | 0.084 | 0.028 | 0 (cut) | 0.066 |

1. $T=0.7$ divides the logits ($\log0.398=-0.92$, $\log0.239=-1.43$) by 0.7 before softmax. That raises cat from 0.398 to 0.504 and shrinks the tail from 0.084 to 0.028.
2. Top-p 0.9: sorted cumulative mass is 0.398, 0.637, 0.776, 0.916. The fourth token crosses 0.9, so four survive and are divided by 0.916.
3. Frequency penalty: "cat" occurred once and "the" twice, so their logits drop by 0.6 and 1.2. Cat falls to $-1.52$, below dog's $-1.43$, and greedy now picks dog. Without the penalty greedy picks cat again and loops.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("gpt2-large")
lm = AutoModelForCausalLM.from_pretrained("gpt2-large").eval()
ids = tok("The cat sat on the", return_tensors="pt").input_ids

rules = {
    "greedy":  dict(do_sample=False),
    "beam":    dict(do_sample=False, num_beams=4, no_repeat_ngram_size=3),
    "nucleus": dict(do_sample=True, top_p=0.9, temperature=1.0, top_k=0),
    "top-k":   dict(do_sample=True, top_k=40, temperature=0.7),
    "penalty": dict(do_sample=True, top_p=0.9, repetition_penalty=1.2),
}
for name, kw in rules.items():
    torch.manual_seed(0)
    out = lm.generate(ids, max_new_tokens=40, pad_token_id=tok.eos_token_id, **kw)
    print(name, "|", tok.decode(out[0, ids.shape[1]:]))

@torch.no_grad()
def perplexity(text):                       # exp of mean next-token cross-entropy
    x = tok(text, return_tensors="pt").input_ids
    return torch.exp(lm(x, labels=x).loss).item()   # labels are shifted inside the model
```

Comparing decoding rules with Hugging Face generate, and scoring a text's perplexity (not run here)

> **Low perplexity of the output is not quality:** Greedy and beam search output has *lower* perplexity than human text under the model that produced it. In the figure, the loop scores 1.83 against the human sentence's 2.60. Choosing a decoding rule by the perplexity of its own outputs therefore selects degenerate text. Compare against human text with distributional measures (MAUVE, repetition rates) and human judgments.

> **Closed vs open-ended tasks:** Maximisation suits tasks where one output is right, such as translation, summarization or extracting an answer. There beam search with length normalisation works well, though a very wide beam still finds bad short outputs. Open-ended tasks need sampling from a truncated distribution. Many chat systems sample with $T\approx0.7$–$1$ and top-p $0.9$–$0.95$, and use $T=0$ (greedy) when reproducibility matters.

**Question:** Why does nucleus sampling adapt better than top-k with a fixed k?

- A. Its candidate set grows when the distribution is flat and shrinks when it is peaked
- B. It always keeps exactly p·|V| tokens
- C. It removes the need for a softmax
- D. It is deterministic, so outputs are reproducible

**Answer:** A. Its candidate set grows when the distribution is flat and shrinks when it is peaked — $V^{(p)}$ is the smallest set with mass $\ge p$. After "The capital of France is", one token may hold 0.95, so the nucleus has one token. After "My favourite food is", hundreds of tokens share the mass and the nucleus grows. A fixed $k$ is too wide in the first case and too narrow in the second.

**Question:** In the worked example, the logit of 'cat' is −0.92 and it has appeared once. With a frequency penalty α = 0.6, what is its new logit, and does greedy still pick it over 'dog' (−1.43)?

- A. −1.52; no, dog wins
- B. −0.32; yes
- C. −1.52; yes, cat still wins
- D. −0.55; yes

**Answer:** A. −1.52; no, dog wins — $-0.92-0.6\cdot1=-1.52<-1.43$, so dog now has the highest logit (0.303 vs 0.277 after softmax). A penalty proportional to the count is what breaks the "the cat sat on the cat …" loop.

**Key points**

- Generation = model $\prod_t p(x_t\mid x_{<t})$ + decoding rule. Training uses teacher forcing; inference feeds back the model's own tokens.
- Greedy and beam search maximise likelihood and degenerate into loops in open-ended text. Human text is not the most probable text.
- Top-$k$ keeps $k$ tokens; nucleus keeps the smallest set with mass $\ge p$, which adapts to the model's confidence. Temperature rescales logits first.
- Penalties edit logits from the history ($z_j-\alpha c_j$, CTRL's $\div1.2$, no-repeat $n$-grams); unlikelihood training addresses repetition during training.
- Control: $p(x_t\mid x_{<t},a)\propto p(x_t\mid x_{<t})\,p(a\mid x_{\le t})$ (FUDGE), control codes (CTRL), or instructions.
- Evaluate with perplexity (the model), distinct-n/rep-n (loops), MAUVE (distribution match) and human judgments.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
