Definition
Logit bias is a per-token offset added to the model's next-token logits before sampling: a map from token ID to a number . Positive values make a token more likely, negative values less likely, and very large magnitudes ban it or force a choice among the biased tokens. The OpenAI API exposes it as logit_bias, with values from −100 to 100; Hugging Face transformers, vLLM and llama.cpp offer equivalents.
Prompts ask; logit bias enforces, at the level of single tokens. Typical jobs are banning a word or a formatting token, nudging a label up or down, and turning a chat model into a classifier that can only emit one of label tokens. It costs nothing at inference time and needs no access to the weights. That is why it was one of the first steering knobs offered by hosted APIs.
It sits in the same logits-processing stage as Temperature sampling, Top-k and top-p sampling and Repetition and frequency penalties. Constrained decoding is the general version: it recomputes a mask at every step from a grammar or schema. Logit bias is a static map applied identically at every step.
The mechanism
- the model's logit for token ID
- the bias for that ID; 0 for every token not in the map
- next-token probabilities before and after the bias
For a single biased token the effect is exact in log-odds: the bias adds to . In probability it depends on where you start, which is why the same helps a rare token far more in relative terms than a common one.
- the odds multiplier: , ,
So −100 is a ban in practice and +100 makes a token beat any unbiased competitor. If several tokens all get +100, they compete on their original logits: the output is restricted to that set at the model's own relative odds.
Interfaces (as documented in 2024)
- OpenAI API
logit_bias: JSON object mapping token ID (as a string key) to a bias in [−100, 100]; the docs say ±1 shifts likelihood slightly and ±100 should ban or force a token- Token IDs
- come from the model's tokenizer, e.g.
tiktokenencodingscl100k_base(GPT-3.5/GPT-4) ando200k_base(GPT-4o) - Hugging Face
sequence_biasmaps token-ID tuples to biases;bad_words_idsbans sequences;suppress_tokenssets logits to −∞- llama.cpp
--logit-bias TOKEN_ID+BIAS(or-infto ban) on the command line- Scope
- the map is applied at every generated position, not just the first
Worked example
Start from the viz's distribution, and .
| Change | Token | p | Computation | p′ |
|---|---|---|---|---|
| b = +2 on ␣neutral | ␣neutral | 0.0468 | 0.0468 × 7.389 / (1 − 0.0468 + 0.0468 × 7.389) | 0.266 |
| b = +2 on ␣neutral | ␣positive | 0.2833 | 0.2833 / 1.299 | 0.218 |
| b = −100 on ␣positive | ␣mixed | 0.1899 | 0.1899 × 1.395 | 0.265 |
| b = −100 on ␣positive | ␣Positive | 0.1042 | 0.1042 × 1.395 | 0.145 |
| +100 on 3 labels | ␣positive | 0.2833 | e³ / (e³ + e^1.8 + e^1.2) | 0.682 |
1import tiktoken2from openai import OpenAI34MODEL = "gpt-4o-mini"5enc = tiktoken.encoding_for_model(MODEL) # o200k_base67labels = [" positive", " negative", " neutral"] # leading space: as it follows "Sentiment:"8ids = [enc.encode(l) for l in labels]9assert all(len(t) == 1 for t in ids), "each label must be a single token"1011resp = OpenAI().chat.completions.create(12 model=MODEL,13 messages=[{"role": "user", "content":14 "Review: Great food, slow service.\nSentiment:"}],15 max_tokens=1, # bias applies at every step16 logit_bias={t[0]: 100 for t in ids},17 logprobs=True, top_logprobs=3, # see the odds, not just the label18)19print(resp.choices[0].message.content)Where it's used
Closed-set classification
+100 on single-token labels and max_tokens 1; read the label probabilities from logprobs
Banning
−100 on a competitor's name, a profanity list or an unwanted formatting token (every spelling)
Soft steering
±1 to ±5 to make a style word or label more or less likely without forbidding it
Check yourself
A token has p = 0.10. What is its probability after a bias of b = ln 3 ≈ 1.10 on that token alone?
Check yourself
You set +100 on the IDs of ' yes' and ' no'. The model's logits for them are 2.0 and 1.0. What are their probabilities?
Remember
- ✓ before softmax; a single token's log-odds shift by exactly .
- ✓−100 bans (); +100 on a set restricts output to that set at its original odds.
- ✓It is keyed by token ID: encode every spelling with the model's own tokenizer.
- ✓The map applies at every step; pair forcing with max_tokens 1.
- ✓For formats that change token by token, use constrained decoding instead.