Temperature sampling

Temperature divides the logits by T before softmax: p ∝ exp(z/T). T < 1 sharpens towards greedy argmax, T > 1 flattens towards uniform, and the token ranking never changes; only how often the tail is sampled does.

7 min read555 wordscore0 viewsupdated 2026-10-21

Definition

Temperature sampling draws each next token from softmax⁡(z/T)\operatorname{softmax}(z/T), where zz are the model's logits and T>0T>0 is a scalar set by the caller. The name and the formula come from the Boltzmann distribution of statistical mechanics, p∝e−E/kTp\propto e^{-E/kT}, with the negative logit playing the role of energy. The same knob softens teacher outputs in Knowledge distillation (Hinton, Vinyals and Dean, 2015) and calibrates classifiers in temperature scaling (Guo et al., 2017).

A language model outputs one logit per vocabulary entry at every step (see Language modeling). The decoder must turn those logits into one token. Taking the argmax (Greedy and beam search) is deterministic and tends to repeat itself on open-ended text. Sampling from the raw softmax is diverse but occasionally picks a token from the long, low-quality tail.

Temperature is the simplest dial between those two failures. It is the first logits processor in most stacks: penalties and biases adjust zz, temperature rescales it, then truncation (Top-k and top-p sampling, Min-p sampling) cuts the tail, and finally one token is drawn. Every hosted LLM API exposes it as temperature.

The formula

pi(T)=exp⁡(zi/T)∑j=1∣V∣exp⁡(zj/T),pi(T)pj(T)=exp⁡ ⁣(zi−zjT)p_i(T)=\frac{\exp(z_i/T)}{\sum_{j=1}^{|V|}\exp(z_j/T)},\qquad \frac{p_i(T)}{p_j(T)}=\exp\!\left(\frac{z_i-z_j}{T}\right)
ziz_i
logit of token i at the current step (unnormalised log-probability)
TT
temperature; T = 1 is the model's own distribution
∣V∣|V|
vocabulary size, typically tens of thousands to a few hundred thousand tokens

The ratio form says everything. A logit gap of Δ\Delta becomes a probability ratio of eΔ/Te^{\Delta/T}. Halving TT squares every ratio; doubling TT takes its square root. Because TT scales all gaps equally, the order of tokens never changes: temperature cannot promote a token above one the model scored higher.

The two limits: as T→0+T\to0^+ all mass goes to the argmax, so APIs treat T=0T=0 as greedy decoding. As T→∞T\to\infty every ratio tends to 1 and the distribution becomes uniform over the whole vocabulary, junk included.

one decoding step: the same logits at three temperatureslogits z2.0, 1.0, 0.5, −1.0divide by Tz / Tsoftmaxe^(z/T) / Σ e^(z/T)sampledraw u ~ U(0,1)T = 0.5H = 0.78 bitsz/Tpsunny4.000.842cloudy2.000.114rainy1.000.042purple−2.000.002T = 1H = 1.46 bitsz/Tpsunny2.000.609cloudy1.000.224rainy0.500.136purple−1.000.030T = 2H = 1.82 bitsz/Tpsunny1.000.434cloudy0.500.263rainy0.250.205purple−0.500.097T < 1 sharpens (sunny 0.609 → 0.842), T > 1 flattens (purple 0.030 → 0.097). Order never changes.T → 0: argmax (greedy) · T → ∞: uniform · p_i / p_j = exp((z_i − z_j) / T)
fig.One decoding step with logits 2.0, 1.0, 0.5, −1.0. At T = 0.5 the logits double, the top token gets 0.842 and entropy is 0.78 bits. At T = 1 it is 0.609 and 1.46 bits. At T = 2 the top token falls to 0.434, the off-topic token rises from 0.030 to 0.097, and entropy is 1.82 bits. The ranking is the same in all three panels.

Measuring the effect: entropy

H(T)=−∑ipi(T)log⁡2pi(T),0≤H(T)≤log⁡2∣V∣H(T)=-\sum_i p_i(T)\log_2 p_i(T),\qquad 0\le H(T)\le\log_2|V|
H(T)H(T)
entropy of the next-token distribution in bits; it increases monotonically with T unless all logits are equal
2H2^{H}
perplexity of the step: the number of equally likely tokens with the same uncertainty

Temperature changes the whole distribution, not just the head. Mass that leaves the top token is spread over every other token in proportion to ezi/Te^{z_i/T}. With a vocabulary of 100k tokens, a small per-token gain in the tail adds up to a real chance of sampling nonsense, and over hundreds of steps one bad token can derail the rest of the text. That is why high temperature is usually paired with truncation.

Where T is set (documented API behaviour as of 2024–2025)

OpenAI Chat Completions
temperature from 0 to 2, default 1; the docs advise changing temperature or top_p, not both
Anthropic Messages API
temperature from 0 to 1, default 1
Hugging Face transformers
do_sample=False (greedy) by default; with do_sample=True, temperature defaults to 1.0 and must be strictly positive
Codex (Chen et al., 2021)
on HumanEval the best temperature was 0.2 for pass@1 and 0.8 for pass@100: more samples reward more diversity (see Code generation)
T = 0 in practice
close to deterministic but not guaranteed: batched GPU kernels can change floating-point sums, which flips near-ties between tokens
"The weather today is …" · p = softmax(z / T), T = 0.70p at T (bar) · p at T = 1 (tick)pdrawssunny0.44249cloudy0.21618warm0.14114cold0.0805rainy0.0606nice0.0343perfect0.0122windy0.0092gloomy0.0051purple0.0000banana0.0000seventeen0.0000rose = off-topic tokens · draws: 100 samples from p at T, seed 1uniform: log₂12 = 3.58 bits00.511.52entropy (bits) vs temperature Tp(sunny)0.442entropy2.28 bitseffective choices 2^H4.86sunny : cloudy2.04 : 1off-topic mass0.0006P(≥1 off-topic in 100)0.059100 draws: distinct, off-topic9, 0
draws
fig.Twelve logits for "The weather today is …", three of them off-topic (purple, banana, seventeen). At the default T = 0.7: p(sunny) = 0.442, entropy 2.28 bits (4.86 effective choices), sunny : cloudy = 2.04 : 1, off-topic mass 0.0006, so P(at least one off-topic token in 100 draws) = 0.059; seed 1 gives 9 distinct tokens and none off-topic. At T = 1: 0.336, 2.67 bits, off-topic mass 0.0045, P = 0.361. At T = 2: 0.206, 3.19 bits, off-topic mass 0.0404, P = 0.984, and the 100 draws hit 12 distinct tokens, 6 of them off-topic. T = 0 gives 100 × sunny.

Worked example

Logits z=(2.0, 1.0, 0.5, −1.0)z=(2.0,\,1.0,\,0.5,\,-1.0) for sunny, cloudy, rainy, purple. Subtract the largest scaled logit before exponentiating so nothing overflows.

Tokenzz/0.5e^(z/0.5 − 4)p at T = 0.5z/2e^(z/2 − 1)p at T = 2
sunny2.0410.8421.0010.434
cloudy1.020.13530.1140.500.60650.263
rainy0.510.04980.0420.250.47240.205
purple−1.0−20.00250.002−0.500.22310.097
sum1.187612.30201
softmax(z / T) by hand, for T = 0.5 and T = 2 (T = 1 gives 0.609, 0.224, 0.136, 0.030)

The sunny : cloudy gap of 1.0 becomes a ratio of e2=7.389e^{2}=7.389 at T=0.5T=0.5 and e0.5=1.649e^{0.5}=1.649 at T=2T=2. Entropy goes from 0.78 bits (1.71 effective choices) to 1.82 bits (3.54 effective choices).

python
1import numpy as np
2
3def softmax_t(logits, T):
4 """Next-token distribution at temperature T; T = 0 means greedy (one-hot argmax)."""
5 z = np.asarray(logits, dtype=np.float64)
6 if T == 0:
7 p = np.zeros_like(z); p[z.argmax()] = 1.0
8 return p
9 z = (z - z.max()) / T # subtract the max first: exp never overflows
10 p = np.exp(z)
11 return p / p.sum()
12
13def entropy_bits(p):
14 p = p[p > 0]
15 return float(-(p * np.log2(p)).sum()) + 0.0 # + 0.0 turns -0.0 into 0.0
16
17z = [2.0, 1.0, 0.5, -1.0] # sunny, cloudy, rainy, purple
18for T in (0, 0.5, 1.0, 2.0):
19 p = softmax_t(z, T)
20 print(T, p.round(3), round(entropy_bits(p), 2))
21# 0 [1. 0. 0. 0.] 0.0
22# 0.5 [0.842 0.114 0.042 0.002] 0.78
23# 1.0 [0.609 0.224 0.136 0.03 ] 1.46
24# 2.0 [0.434 0.263 0.205 0.097] 1.82
25
26rng = np.random.default_rng(0)
27token = rng.choice(len(z), p=softmax_t(z, 0.7)) # one sampled token id
Temperature softmax with T = 0 as greedy, entropy, and one sampled token (numpy; runs as is)
TaskTypical TWhy
Extraction, classification, Text-to-SQL0–0.2one right answer; diversity only adds errors
Code, single attempt0–0.2matches the Codex pass@1 result
Chat assistants, summaries0.5–1.0natural phrasing without drifting
Brainstorming, fiction, many samples0.8–1.2 with truncationdiversity is the goal; truncation guards the tail
Self-consistency voting> 0votes need different reasoning paths; T = 0 gives identical samples
Common starting points, not rules; tune on your own evaluation set

Check yourself

Two tokens have logits 3.0 and 1.0. What is their probability ratio at T = 0.5?

Check yourself

Can raising the temperature make the model's second-ranked token more likely than its first-ranked one?

Remember

  • ✓pi∝exp⁡(zi/T)p_i\propto\exp(z_i/T): a logit gap Δ\Delta becomes a probability ratio eΔ/Te^{\Delta/T}.
  • ✓T → 0 is greedy argmax, T = 1 is the model's distribution, T → ∞ is uniform; token order never changes.
  • ✓Entropy rises with T, and the extra mass is spread over the entire tail, which is why high T needs truncation.
  • ✓APIs: OpenAI 0–2, Anthropic 0–1, both default 1; T = 0 is near-deterministic, not guaranteed.
  • ✓Low T for single-answer tasks, higher T when you sample many candidates and pick or vote.
← back to Generative AI roadmap