Logit bias

Logit bias adds a fixed value b to chosen token IDs' logits before softmax, shifting each token's log-odds by exactly b; −100 bans a token, +100 on a set restricts output to it, but it works on token IDs, not words.

7 min read469 wordscore0 viewsupdated 2026-10-21

Definition

Logit bias is a per-token offset added to the model's next-token logits before sampling: a map from token ID to a number bb. Positive values make a token more likely, negative values less likely, and very large magnitudes ban it or force a choice among the biased tokens. The OpenAI API exposes it as logit_bias, with values from −100 to 100; Hugging Face transformers, vLLM and llama.cpp offer equivalents.

Prompts ask; logit bias enforces, at the level of single tokens. Typical jobs are banning a word or a formatting token, nudging a label up or down, and turning a chat model into a classifier that can only emit one of kk label tokens. It costs nothing at inference time and needs no access to the weights. That is why it was one of the first steering knobs offered by hosted APIs.

It sits in the same logits-processing stage as Temperature sampling, Top-k and top-p sampling and Repetition and frequency penalties. Constrained decoding is the general version: it recomputes a mask at every step from a grammar or schema. Logit bias is a static map applied identically at every step.

The mechanism

zi′=zi+bi⟹pi′=pi ebi∑jpj ebjz'_i=z_i+b_i\qquad\Longrightarrow\qquad p'_i=\frac{p_i\,e^{b_i}}{\sum_j p_j\,e^{b_j}}
ziz_i
the model's logit for token ID ii
bib_i
the bias for that ID; 0 for every token not in the map
p, p′p,\ p'
next-token probabilities before and after the bias

For a single biased token the effect is exact in log-odds: the bias adds bb to log⁡p1−p\log\frac{p}{1-p}. In probability it depends on where you start, which is why the same +2+2 helps a rare token far more in relative terms than a common one.

logit⁡(p′)=logit⁡(p)+b  ⟺  p′=p eb1−p+p eb\operatorname{logit}(p')=\operatorname{logit}(p)+b\quad\iff\quad p'=\frac{p\,e^{b}}{1-p+p\,e^{b}}
ebe^{b}
the odds multiplier: e1=2.72e^{1}=2.72, e2=7.39e^{2}=7.39, e−100≈3.7×10−44e^{-100}\approx3.7\times10^{-44}

So −100 is a ban in practice and +100 makes a token beat any unbiased competitor. If several tokens all get +100, they compete on their original logits: the output is restricted to that set at the model's own relative odds.

POST /v1/chat/completions{ "messages": [{"role": "user", "content": "Review: Great food, slow service. Sentiment:"}], "max_tokens": 1, "logit_bias": { "1001": 100, "1002": 100, "1003": 100 }}← ␣positive← ␣negative← ␣neutraltoy IDs: look real ones up with themodel's own tokenizerp′ = softmax(z + b)tokenzbpp′␣positive3.0+100.283.682␣mixed2.60.1900⏎2.20.1270␣Positive2.00.1040␣negative1.8+100.085.205␣neutral1.2+100.047.1134 others0.1630+100 on all three = keep only them, at theiroriginal odds: e³ : e^1.8 : e^1.2−100 multiplies odds by e⁻¹⁰⁰ ≈ 3.7 × 10⁻⁴⁴: a banlogit bias acts on token IDs, not words (illustrative splits)“positive”␣positive · positive · ␣Positiveone word, three IDs: ban all or none“unhappy”␣un + happybiasing ␣un hits every ␣un… word“Paris”IDs differ across tokenizersa bias map is tied to one model's vocabulary
fig.A one-token classifier. The request adds +100 to the token IDs of ␣positive, ␣negative and ␣neutral (toy IDs). Before the bias the model splits its mass across ten tokens, with ␣mixed (0.190) second and a newline third. After it, only the labels survive, at their original odds e³ : e^1.8 : e^1.2 = 0.682 : 0.205 : 0.113. Below: the bias acts on token IDs, so one word can need several entries, a word piece hits every word that shares it, and a bias map belongs to one tokenizer.

Interfaces (as documented in 2024)

OpenAI API
logit_bias: JSON object mapping token ID (as a string key) to a bias in [−100, 100]; the docs say ±1 shifts likelihood slightly and ±100 should ban or force a token
Token IDs
come from the model's tokenizer, e.g. tiktoken encodings cl100k_base (GPT-3.5/GPT-4) and o200k_base (GPT-4o)
Hugging Face
sequence_bias maps token-ID tuples to biases; bad_words_ids bans sequences; suppress_tokens sets logits to −∞
llama.cpp
--logit-bias TOKEN_ID+BIAS (or -inf to ban) on the command line
Scope
the map is applied at every generated position, not just the first
Review: “Great food, slow service.” Sentiment:p′ ∝ exp(z + b) · ␣ = leading space · click a row to edit its biastokenzbias bp (outline) → p′ (filled)p → p′␣positive ◆3.000.283 → 0.283␣Positive2.000.104 → 0.104␣mixed2.600.190 → 0.190␣negative ◆1.800.085 → 0.085␣neutral ◆1.200.047 → 0.047␣Mixed1.000.038 → 0.038␣pos0.500.023 → 0.023␣The1.500.063 → 0.063⏎2.200.127 → 0.127␣It1.000.038 → 0.038argmax ␣positive · “positive” in any spelling 41.1% → 41.1% · labels ◆ 41.5% → 41.5%␣neutral: b = 0 multiplies its odds by e^b = 1.000a bias is added to the logit, so its effect on p depends on the other tokens
preset
␣neutral
fig.Ten next-token candidates after a sentiment prompt; click a row to set its bias. No bias: ␣positive is the argmax at 0.283 and the three labels hold 41.5% of the mass. Ban ␣positive: the argmax becomes ␣mixed (0.265), yet 17.8% still goes to ␣Positive and ␣pos. Labels only (+100 ×3): ␣positive 0.682, ␣negative 0.205, ␣neutral 0.113. Nudge ␣neutral +2: 0.047 → 0.266, and it becomes the argmax.

Worked example

Start from the viz's distribution, p(␣neutral)=0.0468p(\text{␣neutral})=0.0468 and p(␣positive)=0.2833p(\text{␣positive})=0.2833.

ChangeTokenpComputationp′
b = +2 on ␣neutral␣neutral0.04680.0468 × 7.389 / (1 − 0.0468 + 0.0468 × 7.389)0.266
b = +2 on ␣neutral␣positive0.28330.2833 / 1.2990.218
b = −100 on ␣positive␣mixed0.18990.1899 × 1.3950.265
b = −100 on ␣positive␣Positive0.10420.1042 × 1.3950.145
+100 on 3 labels␣positive0.2833e³ / (e³ + e^1.8 + e^1.2)0.682
Every number from p′ = p·e^b / Σ p·e^b; a ban rescales the rest by 1/(1 − 0.2833) = 1.395
python
1import tiktoken
2from openai import OpenAI
3
4MODEL = "gpt-4o-mini"
5enc = tiktoken.encoding_for_model(MODEL) # o200k_base
6
7labels = [" positive", " negative", " neutral"] # leading space: as it follows "Sentiment:"
8ids = [enc.encode(l) for l in labels]
9assert all(len(t) == 1 for t in ids), "each label must be a single token"
10
11resp = OpenAI().chat.completions.create(
12 model=MODEL,
13 messages=[{"role": "user", "content":
14 "Review: Great food, slow service.\nSentiment:"}],
15 max_tokens=1, # bias applies at every step
16 logit_bias={t[0]: 100 for t in ids},
17 logprobs=True, top_logprobs=3, # see the odds, not just the label
18)
19print(resp.choices[0].message.content)
Restricting a chat model to three label tokens with tiktoken and the OpenAI Python SDK (not run here: neither package was available; the model name is a 2024 example)

Where it's used

Closed-set classification

+100 on single-token labels and max_tokens 1; read the label probabilities from logprobs

Banning

−100 on a competitor's name, a profanity list or an unwanted formatting token (every spelling)

Soft steering

±1 to ±5 to make a style word or label more or less likely without forbidding it

Check yourself

A token has p = 0.10. What is its probability after a bias of b = ln 3 ≈ 1.10 on that token alone?

Check yourself

You set +100 on the IDs of ' yes' and ' no'. The model's logits for them are 2.0 and 1.0. What are their probabilities?

Remember

  • ✓zi′=zi+biz'_i=z_i+b_i before softmax; a single token's log-odds shift by exactly bb.
  • ✓−100 bans (e−100≈3.7×10−44e^{-100}\approx3.7\times10^{-44}); +100 on a set restricts output to that set at its original odds.
  • ✓It is keyed by token ID: encode every spelling with the model's own tokenizer.
  • ✓The map applies at every step; pair forcing with max_tokens 1.
  • ✓For formats that change token by token, use constrained decoding instead.
← back to Generative AI roadmap