# Repetition and frequency penalties

> Repetition penalties lower the logits of tokens already in the text: CTRL divides them by θ ≈ 1.2 (Keskar et al., 2019), while OpenAI-style penalties subtract α_f·count + α_p·[seen], so only the frequency term grows with every repeat.

Source: https://www.mlfoundry.in/learn/repetition-and-frequency-penalties  
Section: Generative AI › Decoding and Generation Control  
Level: core · 7 min read · updated 2026-10-21

**Repetition penalty:** is a decode-time change to the next-token logits that makes tokens already present in the context less likely. The multiplicative form comes from CTRL (Keskar, McCann, Varshney, Xiong and Socher, "CTRL: A Conditional Transformer Language Model for Controllable Generation", 2019). The additive **frequency** and **presence** penalties are the form exposed by the OpenAI API and copied by many OpenAI-compatible servers.

Greedy and low-temperature decoding fall into loops. Once a phrase has been generated, the model's own output makes the same continuation more likely, and the text repeats a sentence until the token limit. Holtzman et al. ("The Curious Case of Neural Text Degeneration", ICLR 2020) documented this for greedy and beam search. Small models, long outputs and near-greedy settings make it worse.

Penalties are the cheapest fix: they need no retraining and sit in the logits-processing stage, next to **Temperature sampling** and **Top-k and top-p sampling**. They work on token identities, not meaning. A penalty cannot tell a stuck loop from a name, a keyword or a closing bracket that has to appear again.

## The two families

$$
p_i=\frac{\exp\!\big(z_i/(T\cdot I(i\in g))\big)}{\sum_j\exp\!\big(z_j/(T\cdot I(j\in g))\big)},\qquad I(c)=\begin{cases}\theta & c\text{ true}\\ 1&\text{otherwise}\end{cases}
$$

- $g$: the tokens generated so far (Hugging Face counts the whole input, prompt included)
- $\theta$: repetition penalty; CTRL found $\theta\approx1.2$ a good balance, $\theta=1$ disables it
- $T$: temperature; CTRL paired the penalty with near-greedy sampling

Dividing a positive logit by $\theta>1$ lowers it, but dividing a negative one raises it. Hugging Face's `repetition_penalty` therefore divides positive logits and multiplies negative ones, so a seen token always loses. Either way the penalty is the same whether the token appeared once or fifty times.

$$
z'_j=z_j-\alpha_\text{f}\,c_j-\alpha_\text{p}\,\mathbb 1[c_j>0]\quad\Longrightarrow\quad\frac{p'_j}{p'_k}=\frac{p_j}{p_k}\,e^{-\alpha_\text{f}c_j-\alpha_\text{p}\mathbb 1[c_j>0]}\ \ (c_k=0)
$$

- $c_j$: how many times token $j$ has been sampled so far
- $\alpha_\text{f}$: frequency penalty: grows linearly with each repeat
- $\alpha_\text{p}$: presence penalty: one-off, applied once a token has appeared at all

The additive form has a clean reading: each repeat multiplies the token's odds against an unseen token by $e^{-\alpha_\text{f}}$. With $\alpha_\text{f}=0.5$ that is a factor 0.61 per occurrence, and 0.082 after five occurrences. Negative values are allowed and encourage repetition.

> Figure: Penalties act between the logits and the softmax. After “the cat sat on the”, the unpenalised model picks cat again (p = 0.454) and the loop continues. CTRL θ = 1.2 lowers cat to 3.33 and frequency + presence 0.5 lowers it to 3.00; both now pick dog. Right: the frequency penalty grows with every repeat, while presence and CTRL are flat once a token is seen; CTRL takes 0.67 from z = 4 but only 0.17 from z = 1.

Common knobs (names as in each library's documentation; check your version)

| Where | Parameter | Form | Neutral value |
| --- | --- | --- | --- |
| Hugging Face generate | repetition_penalty | CTRL θ, sign-aware | 1.0 |
| Hugging Face generate | no_repeat_ngram_size | hard ban on any repeated n-gram | 0 (off) |
| OpenAI API (chat/completions) | frequency_penalty, presence_penalty | additive, each in [−2, 2] | 0 |
| vLLM SamplingParams | repetition_penalty, frequency_penalty, presence_penalty | both families | 1.0, 0, 0 |

**Documented guidance**

- **CTRL (2019):** θ ≈ 1.2 with near-greedy sampling gave a good balance between truthful generation and lack of repetition
- **OpenAI docs (2023–24):** values around 0.1–1 reduce repetition somewhat; up to 2 suppresses it strongly but can noticeably degrade quality
- **n-gram blocking:** trigram blocking at test time was used in abstractive summarisation (Paulus et al., 2018); it is a hard rule, not a soft penalty
- **Training-side fix:** unlikelihood training (Welleck et al., ICLR 2020) penalises repeats during training instead of at decode time

> Figure: A toy bigram model decoded greedily for 36 steps from “the”. No penalty: “the cat sat on” nine times over, distinct-2 0.111, “the” ×10. CTRL θ = 1.2 escapes once (“dog ran .”) and falls back (0.222); θ = 2 or presence 2.0 escape further but still return to the loop once every alternative has been seen (0.500). Default frequency 0.5 + presence 0.5: distinct-2 0.583, mean unpenalised log p −1.053. Frequency 2.0 alone: 0.722, log p −1.397, one implausible bigram (“slept mat”).

The viz shows the central difference. A flat penalty (CTRL or presence) only reorders seen against unseen tokens. When every plausible continuation has been used once, the original ranking returns and so does the loop. A frequency penalty keeps rising, so the loop's tokens fall further behind every time it comes round. Push it too far and fluency pays: the mean log-probability under the unpenalised model drops and implausible tokens start to win.

## Worked example

The text so far is "the cat sat on the", so cat has count 1 and the has count 2. Five candidates follow "the"; probabilities are a softmax over these five.

CTRL θ = 1.2 and OpenAI-style α_f = α_p = 0.5 both move the argmax from cat to dog

| Token | c | z (p) | CTRL θ = 1.2 | z − 0.5c − 0.5·[c > 0] |
| --- | --- | --- | --- | --- |
| cat | 1 | 4.00 (0.454) | 4.00 / 1.2 = 3.33 (0.299) | 4.00 − 0.5 − 0.5 = 3.00 (0.235) |
| dog | 0 | 3.50 (0.275) | 3.50 (0.353) | 3.50 (0.387) |
| mat | 0 | 3.00 (0.167) | 3.00 (0.214) | 3.00 (0.235) |
| rug | 0 | 2.50 (0.101) | 2.50 (0.130) | 2.50 (0.142) |
| the | 2 | −1.00 (0.003) | −1.00 × 1.2 = −1.20 (0.003) | −1.00 − 1.0 − 0.5 = −2.50 (0.001) |

A naive CTRL implementation would compute $-1.00/1.2=-0.83$ for "the", which raises a token it is meant to suppress. The sign-aware rule avoids that.

```python
import numpy as np

def apply_penalties(logits, history, repetition=1.0, frequency=0.0, presence=0.0):
    """CTRL/HF repetition penalty, then OpenAI-style frequency/presence penalties."""
    z = logits.astype(float).copy()
    counts = np.bincount(history, minlength=len(z))
    seen = counts > 0
    z[seen] = np.where(z[seen] > 0, z[seen] / repetition, z[seen] * repetition)
    return z - frequency * counts - presence * seen

np.set_printoptions(precision=3)
vocab = ["cat", "dog", "mat", "rug", "the"]
logits = np.array([4.0, 3.5, 3.0, 2.5, -1.0])
history = [4, 0, 4]                     # ids in "the cat ... the": cat 1, the 2
print(apply_penalties(logits, history, repetition=1.2))               # [ 3.333  3.5    3.     2.5   -1.2  ]
print(apply_penalties(logits, history, frequency=0.5, presence=0.5))  # [ 3.   3.5  3.   2.5 -2.5]
```

Both penalty families on the worked example (numpy, runs as is). The OpenAI API takes the same numbers as frequency_penalty and presence_penalty in the request body.

> **Penalties do not know what must repeat:** Code, JSON, tables and legal text need the same identifiers, keys and brackets many times. A frequency penalty taxes every `"name"` key and every closing brace, and with high values the model starts substituting synonyms, misspelling variable names or breaking the format. For structured output keep penalties at 0 and use **Constrained decoding** or **Structured output (JSON mode)**; for code, prefer a small CTRL θ or none.

> **What counts as "seen":** Implementations differ on whether the prompt counts. Hugging Face's `repetition_penalty` penalises every token in `input_ids`, which in a decoder-only model includes the prompt, so a long document to summarise makes its own vocabulary harder to reuse. Penalties are also per token, not per word: " Paris" and "Paris" are different tokens (see **Byte-pair encoding (BPE)**), and a multi-token word is penalised piece by piece.

- **Small or quantised models:** near-greedy decoding loops quickly; θ ≈ 1.1–1.2 or a light frequency penalty is a common default in local runners
- **Long-form generation:** stories and reports drift into repeated phrases; a frequency penalty of 0.1–0.5 discourages it
- **Brainstorming lists:** a presence penalty pushes the model towards new tokens, which tends to raise topic variety

**Question:** A token has appeared 4 times. With frequency_penalty = 0.3 and presence_penalty = 0.6, how much is subtracted from its logit?

- A. 0.9
- B. 1.2
- C. 1.8
- D. 4.8

**Answer:** C. 1.8 — $0.3\times4+0.6\times1=1.8$. The presence term is applied once, however many times the token occurred.

**Question:** Why can a CTRL-style penalty fail to stop a long loop where a frequency penalty succeeds?

- A. CTRL only applies to the prompt
- B. CTRL's penalty is the same after one occurrence as after fifty, so once all alternatives are seen the original ranking returns
- C. CTRL is applied after sampling
- D. CTRL only works with beam search

**Answer:** B. CTRL's penalty is the same after one occurrence as after fifty, so once all alternatives are seen the original ranking returns — CTRL changes $z_i$ to $z_i/\theta$ once $i\in g$, independent of the count. When the loop's tokens and every alternative have all been seen, they are all penalised alike and the ranking is the unpenalised one. The frequency term $\alpha_\text{f}c_j$ keeps growing with each repeat.

**Key points**

- CTRL (2019): seen tokens get $z/\theta$ (sign-aware in HF), $\theta\approx1.2$; flat in the count.
- OpenAI-style: $z-\alpha_\text{f}c-\alpha_\text{p}\mathbb 1[c>0]$, each in $[-2,2]$; each repeat multiplies the odds by $e^{-\alpha_\text{f}}$.
- Flat penalties break short loops; only the frequency term keeps pushing a long loop away.
- Too much penalty costs fluency and breaks code, JSON and names that must repeat.
- Penalties act on token IDs: check whether your engine counts the prompt.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
