Definition
Temperature sampling draws each next token from , where are the model's logits and is a scalar set by the caller. The name and the formula come from the Boltzmann distribution of statistical mechanics, , with the negative logit playing the role of energy. The same knob softens teacher outputs in Knowledge distillation (Hinton, Vinyals and Dean, 2015) and calibrates classifiers in temperature scaling (Guo et al., 2017).
A language model outputs one logit per vocabulary entry at every step (see Language modeling). The decoder must turn those logits into one token. Taking the argmax (Greedy and beam search) is deterministic and tends to repeat itself on open-ended text. Sampling from the raw softmax is diverse but occasionally picks a token from the long, low-quality tail.
Temperature is the simplest dial between those two failures. It is the first logits processor in most stacks: penalties and biases adjust , temperature rescales it, then truncation (Top-k and top-p sampling, Min-p sampling) cuts the tail, and finally one token is drawn. Every hosted LLM API exposes it as temperature.
The formula
- logit of token i at the current step (unnormalised log-probability)
- temperature; T = 1 is the model's own distribution
- vocabulary size, typically tens of thousands to a few hundred thousand tokens
The ratio form says everything. A logit gap of becomes a probability ratio of . Halving squares every ratio; doubling takes its square root. Because scales all gaps equally, the order of tokens never changes: temperature cannot promote a token above one the model scored higher.
The two limits: as all mass goes to the argmax, so APIs treat as greedy decoding. As every ratio tends to 1 and the distribution becomes uniform over the whole vocabulary, junk included.
Measuring the effect: entropy
- entropy of the next-token distribution in bits; it increases monotonically with T unless all logits are equal
- perplexity of the step: the number of equally likely tokens with the same uncertainty
Temperature changes the whole distribution, not just the head. Mass that leaves the top token is spread over every other token in proportion to . With a vocabulary of 100k tokens, a small per-token gain in the tail adds up to a real chance of sampling nonsense, and over hundreds of steps one bad token can derail the rest of the text. That is why high temperature is usually paired with truncation.
Where T is set (documented API behaviour as of 2024–2025)
- OpenAI Chat Completions
temperaturefrom 0 to 2, default 1; the docs advise changingtemperatureortop_p, not both- Anthropic Messages API
temperaturefrom 0 to 1, default 1- Hugging Face transformers
do_sample=False(greedy) by default; withdo_sample=True,temperaturedefaults to 1.0 and must be strictly positive- Codex (Chen et al., 2021)
- on HumanEval the best temperature was 0.2 for pass@1 and 0.8 for pass@100: more samples reward more diversity (see Code generation)
- T = 0 in practice
- close to deterministic but not guaranteed: batched GPU kernels can change floating-point sums, which flips near-ties between tokens
Worked example
Logits for sunny, cloudy, rainy, purple. Subtract the largest scaled logit before exponentiating so nothing overflows.
| Token | z | z/0.5 | e^(z/0.5 − 4) | p at T = 0.5 | z/2 | e^(z/2 − 1) | p at T = 2 |
|---|---|---|---|---|---|---|---|
| sunny | 2.0 | 4 | 1 | 0.842 | 1.00 | 1 | 0.434 |
| cloudy | 1.0 | 2 | 0.1353 | 0.114 | 0.50 | 0.6065 | 0.263 |
| rainy | 0.5 | 1 | 0.0498 | 0.042 | 0.25 | 0.4724 | 0.205 |
| purple | −1.0 | −2 | 0.0025 | 0.002 | −0.50 | 0.2231 | 0.097 |
| sum | 1.1876 | 1 | 2.3020 | 1 |
The sunny : cloudy gap of 1.0 becomes a ratio of at and at . Entropy goes from 0.78 bits (1.71 effective choices) to 1.82 bits (3.54 effective choices).
1import numpy as np23def softmax_t(logits, T):4 """Next-token distribution at temperature T; T = 0 means greedy (one-hot argmax)."""5 z = np.asarray(logits, dtype=np.float64)6 if T == 0:7 p = np.zeros_like(z); p[z.argmax()] = 1.08 return p9 z = (z - z.max()) / T # subtract the max first: exp never overflows10 p = np.exp(z)11 return p / p.sum()1213def entropy_bits(p):14 p = p[p > 0]15 return float(-(p * np.log2(p)).sum()) + 0.0 # + 0.0 turns -0.0 into 0.01617z = [2.0, 1.0, 0.5, -1.0] # sunny, cloudy, rainy, purple18for T in (0, 0.5, 1.0, 2.0):19 p = softmax_t(z, T)20 print(T, p.round(3), round(entropy_bits(p), 2))21# 0 [1. 0. 0. 0.] 0.022# 0.5 [0.842 0.114 0.042 0.002] 0.7823# 1.0 [0.609 0.224 0.136 0.03 ] 1.4624# 2.0 [0.434 0.263 0.205 0.097] 1.822526rng = np.random.default_rng(0)27token = rng.choice(len(z), p=softmax_t(z, 0.7)) # one sampled token id| Task | Typical T | Why |
|---|---|---|
| Extraction, classification, Text-to-SQL | 0–0.2 | one right answer; diversity only adds errors |
| Code, single attempt | 0–0.2 | matches the Codex pass@1 result |
| Chat assistants, summaries | 0.5–1.0 | natural phrasing without drifting |
| Brainstorming, fiction, many samples | 0.8–1.2 with truncation | diversity is the goal; truncation guards the tail |
| Self-consistency voting | > 0 | votes need different reasoning paths; T = 0 gives identical samples |
Check yourself
Two tokens have logits 3.0 and 1.0. What is their probability ratio at T = 0.5?
Check yourself
Can raising the temperature make the model's second-ranked token more likely than its first-ranked one?
Remember
- ✓: a logit gap becomes a probability ratio .
- ✓T → 0 is greedy argmax, T = 1 is the model's distribution, T → ∞ is uniform; token order never changes.
- ✓Entropy rises with T, and the extra mass is spread over the entire tail, which is why high T needs truncation.
- ✓APIs: OpenAI 0–2, Anthropic 0–1, both default 1; T = 0 is near-deterministic, not guaranteed.
- ✓Low T for single-answer tasks, higher T when you sample many candidates and pick or vote.