# Temperature sampling

> Temperature divides the logits by T before softmax: p ∝ exp(z/T). T < 1 sharpens towards greedy argmax, T > 1 flattens towards uniform, and the token ranking never changes; only how often the tail is sampled does.

Source: https://www.mlfoundry.in/learn/temperature-sampling  
Section: Generative AI › Decoding and Generation Control  
Level: core · 7 min read · updated 2026-10-21

**Temperature sampling:** draws each next token from $\operatorname{softmax}(z/T)$, where $z$ are the model's logits and $T>0$ is a scalar set by the caller. The name and the formula come from the Boltzmann distribution of statistical mechanics, $p\propto e^{-E/kT}$, with the negative logit playing the role of energy. The same knob softens teacher outputs in **Knowledge distillation** (Hinton, Vinyals and Dean, 2015) and calibrates classifiers in temperature scaling (Guo et al., 2017).

A language model outputs one logit per vocabulary entry at every step (see **Language modeling**). The decoder must turn those logits into one token. Taking the argmax (**Greedy and beam search**) is deterministic and tends to repeat itself on open-ended text. Sampling from the raw softmax is diverse but occasionally picks a token from the long, low-quality tail.

Temperature is the simplest dial between those two failures. It is the first logits processor in most stacks: penalties and biases adjust $z$, temperature rescales it, then truncation (**Top-k and top-p sampling**, **Min-p sampling**) cuts the tail, and finally one token is drawn. Every hosted LLM API exposes it as `temperature`.

## The formula

$$
p_i(T)=\frac{\exp(z_i/T)}{\sum_{j=1}^{|V|}\exp(z_j/T)},\qquad \frac{p_i(T)}{p_j(T)}=\exp\!\left(\frac{z_i-z_j}{T}\right)
$$

- $z_i$: logit of token i at the current step (unnormalised log-probability)
- $T$: temperature; T = 1 is the model's own distribution
- $|V|$: vocabulary size, typically tens of thousands to a few hundred thousand tokens

The ratio form says everything. A logit gap of $\Delta$ becomes a probability ratio of $e^{\Delta/T}$. Halving $T$ squares every ratio; doubling $T$ takes its square root. Because $T$ scales all gaps equally, **the order of tokens never changes**: temperature cannot promote a token above one the model scored higher.

The two limits: as $T\to0^+$ all mass goes to the argmax, so APIs treat $T=0$ as greedy decoding. As $T\to\infty$ every ratio tends to 1 and the distribution becomes uniform over the whole vocabulary, junk included.

> Figure: One decoding step with logits 2.0, 1.0, 0.5, −1.0. At T = 0.5 the logits double, the top token gets 0.842 and entropy is 0.78 bits. At T = 1 it is 0.609 and 1.46 bits. At T = 2 the top token falls to 0.434, the off-topic token rises from 0.030 to 0.097, and entropy is 1.82 bits. The ranking is the same in all three panels.

## Measuring the effect: entropy

$$
H(T)=-\sum_i p_i(T)\log_2 p_i(T),\qquad 0\le H(T)\le\log_2|V|
$$

- $H(T)$: entropy of the next-token distribution in bits; it increases monotonically with T unless all logits are equal
- $2^{H}$: perplexity of the step: the number of equally likely tokens with the same uncertainty

Temperature changes the whole distribution, not just the head. Mass that leaves the top token is spread over **every** other token in proportion to $e^{z_i/T}$. With a vocabulary of 100k tokens, a small per-token gain in the tail adds up to a real chance of sampling nonsense, and over hundreds of steps one bad token can derail the rest of the text. That is why high temperature is usually paired with truncation.

**Where T is set (documented API behaviour as of 2024–2025)**

- **OpenAI Chat Completions:** `temperature` from 0 to 2, default 1; the docs advise changing `temperature` or `top_p`, not both
- **Anthropic Messages API:** `temperature` from 0 to 1, default 1
- **Hugging Face transformers:** `do_sample=False` (greedy) by default; with `do_sample=True`, `temperature` defaults to 1.0 and must be strictly positive
- **Codex (Chen et al., 2021):** on HumanEval the best temperature was 0.2 for pass@1 and 0.8 for pass@100: more samples reward more diversity (see **Code generation**)
- **T = 0 in practice:** close to deterministic but not guaranteed: batched GPU kernels can change floating-point sums, which flips near-ties between tokens

> Figure: Twelve logits for "The weather today is …", three of them off-topic (purple, banana, seventeen). At the default T = 0.7: p(sunny) = 0.442, entropy 2.28 bits (4.86 effective choices), sunny : cloudy = 2.04 : 1, off-topic mass 0.0006, so P(at least one off-topic token in 100 draws) = 0.059; seed 1 gives 9 distinct tokens and none off-topic. At T = 1: 0.336, 2.67 bits, off-topic mass 0.0045, P = 0.361. At T = 2: 0.206, 3.19 bits, off-topic mass 0.0404, P = 0.984, and the 100 draws hit 12 distinct tokens, 6 of them off-topic. T = 0 gives 100 × sunny.

## Worked example

Logits $z=(2.0,\,1.0,\,0.5,\,-1.0)$ for sunny, cloudy, rainy, purple. Subtract the largest scaled logit before exponentiating so nothing overflows.

softmax(z / T) by hand, for T = 0.5 and T = 2 (T = 1 gives 0.609, 0.224, 0.136, 0.030)

| Token | z | z/0.5 | e^(z/0.5 − 4) | p at T = 0.5 | z/2 | e^(z/2 − 1) | p at T = 2 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| sunny | 2.0 | 4 | 1 | 0.842 | 1.00 | 1 | 0.434 |
| cloudy | 1.0 | 2 | 0.1353 | 0.114 | 0.50 | 0.6065 | 0.263 |
| rainy | 0.5 | 1 | 0.0498 | 0.042 | 0.25 | 0.4724 | 0.205 |
| purple | −1.0 | −2 | 0.0025 | 0.002 | −0.50 | 0.2231 | 0.097 |
| sum |  |  | 1.1876 | 1 |  | 2.3020 | 1 |

The sunny : cloudy gap of 1.0 becomes a ratio of $e^{2}=7.389$ at $T=0.5$ and $e^{0.5}=1.649$ at $T=2$. Entropy goes from 0.78 bits (1.71 effective choices) to 1.82 bits (3.54 effective choices).

```python
import numpy as np

def softmax_t(logits, T):
    """Next-token distribution at temperature T; T = 0 means greedy (one-hot argmax)."""
    z = np.asarray(logits, dtype=np.float64)
    if T == 0:
        p = np.zeros_like(z); p[z.argmax()] = 1.0
        return p
    z = (z - z.max()) / T            # subtract the max first: exp never overflows
    p = np.exp(z)
    return p / p.sum()

def entropy_bits(p):
    p = p[p > 0]
    return float(-(p * np.log2(p)).sum()) + 0.0   # + 0.0 turns -0.0 into 0.0

z = [2.0, 1.0, 0.5, -1.0]            # sunny, cloudy, rainy, purple
for T in (0, 0.5, 1.0, 2.0):
    p = softmax_t(z, T)
    print(T, p.round(3), round(entropy_bits(p), 2))
# 0   [1. 0. 0. 0.] 0.0
# 0.5 [0.842 0.114 0.042 0.002] 0.78
# 1.0 [0.609 0.224 0.136 0.03 ] 1.46
# 2.0 [0.434 0.263 0.205 0.097] 1.82

rng = np.random.default_rng(0)
token = rng.choice(len(z), p=softmax_t(z, 0.7))   # one sampled token id
```

Temperature softmax with T = 0 as greedy, entropy, and one sampled token (numpy; runs as is)

Common starting points, not rules; tune on your own evaluation set

| Task | Typical T | Why |
| --- | --- | --- |
| Extraction, classification, **Text-to-SQL** | 0–0.2 | one right answer; diversity only adds errors |
| Code, single attempt | 0–0.2 | matches the Codex pass@1 result |
| Chat assistants, summaries | 0.5–1.0 | natural phrasing without drifting |
| Brainstorming, fiction, many samples | 0.8–1.2 with truncation | diversity is the goal; truncation guards the tail |
| **Self-consistency** voting | > 0 | votes need different reasoning paths; T = 0 gives identical samples |

> **Temperature does not fix facts:** Lowering $T$ makes the model's most likely answer more likely. If that answer is wrong, $T=0$ returns it every time, confidently (see **Hallucination**). Temperature controls diversity, not correctness; grounding and evaluation address correctness.

> **Temperature and truncation are different tools:** Temperature reshapes every probability smoothly and keeps the full vocabulary. Truncation sets most of the tail to exactly zero. High $T$ alone eventually samples junk; truncation alone cannot make a peaked distribution more diverse. Use them together, and remember that the order of operations changes the result (**Top-k and top-p sampling** shows how).

**Question:** Two tokens have logits 3.0 and 1.0. What is their probability ratio at T = 0.5?

- A. 2
- B. e ≈ 2.72
- C. e² ≈ 7.39
- D. e⁴ ≈ 54.6

**Answer:** D. e⁴ ≈ 54.6 — The ratio is $\exp((z_i-z_j)/T)=\exp(2/0.5)=e^4\approx54.6$. At $T=1$ it would be $e^2\approx7.39$: halving the temperature squares the ratio.

**Question:** Can raising the temperature make the model's second-ranked token more likely than its first-ranked one?

- A. Yes, above T = 1 the ranking reverses
- B. No; dividing all logits by the same positive T preserves their order, so ratios move towards 1 but never cross it
- C. Only for vocabularies larger than 50k tokens
- D. Only when top-p is also applied

**Answer:** B. No; dividing all logits by the same positive T preserves their order, so ratios move towards 1 but never cross it — $p_1/p_2=e^{(z_1-z_2)/T}$ stays above 1 for every finite $T>0$ when $z_1>z_2$. It approaches 1 as $T\to\infty$, where the distribution is uniform.

**Key points**

- $p_i\propto\exp(z_i/T)$: a logit gap $\Delta$ becomes a probability ratio $e^{\Delta/T}$.
- T → 0 is greedy argmax, T = 1 is the model's distribution, T → ∞ is uniform; token order never changes.
- Entropy rises with T, and the extra mass is spread over the entire tail, which is why high T needs truncation.
- APIs: OpenAI 0–2, Anthropic 0–1, both default 1; T = 0 is near-deterministic, not guaranteed.
- Low T for single-answer tasks, higher T when you sample many candidates and pick or vote.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
