# Zero-shot prompting

> Zero-shot prompting states the task in words with no examples; read the answer by scoring the label words, ideally minus their content-free prior (PMI), not by parsing free text. GPT-3 175B scored 76.2% zero-shot on LAMBADA.

Source: https://www.mlfoundry.in/learn/zero-shot-prompting  
Section: Generative AI › Prompt Engineering  
Level: core · 7 min read · updated 2026-10-21

**Zero-shot prompting:** is asking a pretrained language model to do a task from a natural-language description alone, with no worked examples in the prompt and no change to the weights. Radford et al. ("Language Models are Unsupervised Multitask Learners", 2019, GPT-2) showed zero-shot task transfer from pretraining alone. Brown et al. ("Language Models are Few-Shot Learners", 2020, GPT-3) fixed the terms: zero-shot gives only an instruction, one-shot adds one demonstration, few-shot adds several.

It is the default way people use a chat assistant, and the first baseline for any new task. It is also the cheapest: the prompt is short, there is nothing to collect or label, and changing the task means editing a sentence. The cost is control. The model must infer the task, the label set and the output format from words alone. When any of these is ambiguous, accuracy drops, and the output may not parse.

A zero-shot prompt has three parts. The **instruction** states the task and the allowed outputs. The **input** is the item to process. The **output cue** is the text right before the answer, such as `Sentiment:`, `A:` or `TL;DR:`, and it tells the model what kind of token comes next. GPT-2 summarised articles zero-shot just by appending `TL;DR:`. In a chat model the instruction usually goes in the system message (see **System prompts and role prompting**), and the cue becomes an explicit format rule.

> Figure: A zero-shot prompt: instruction, input and output cue, with k = 0 demonstrations, fed to a frozen model. The next-token distribution (illustrative) can be read three ways. Free generation decodes and parses text. Label scoring keeps only the label words: 0.42 : 0.18 renormalises to 0.700 / 0.300, so positive. PMI scoring divides by the label prior from the same prompt with the review removed (0.80 / 0.20): 0.875 against 1.500, so negative.

## Reading the answer

For open-ended tasks (summaries, rewrites, answers) you decode text and use it. For classification there is a more reliable option: never let the model choose freely. Score each allowed label instead. A **verbalizer** $v$ maps each class to a label string, such as positive → " positive". The prediction is the class whose string is most likely after the prompt:

$$
\hat y=\arg\max_{y\in\mathcal Y}\;\sum_{t=1}^{|v(y)|}\log P_\theta\big(v(y)_t\mid T(x),\,v(y)_{<t}\big)
$$

- $T(x)$: the prompt template filled with input x (instruction, input, cue)
- $v(y)$: the token sequence of the label for class y; one token in the simple case
- $\mathcal Y$: the closed label set; nothing outside it can be predicted

This removes format failures, because the answer is always a valid label. It does not remove **bias**. Some label words are simply more frequent, so the model prefers them whatever the input says. Labels of several tokens also compete unequally: a summed log-probability penalises longer strings, which is why some evaluations divide by the token count.

Holtzman et al. ("Surface Form Competition: Why the Highest Probability Answer Isn't Always Right", EMNLP 2021) proposed scoring by how much the input *raises* each label's probability. Their domain-conditional PMI divides by the label's probability under a prompt that keeps the task wording but drops the input:

$$
\hat y=\arg\max_{y}\;\Big[\log P_\theta\big(v(y)\mid T(x)\big)-\log P_\theta\big(v(y)\mid T(\varnothing)\big)\Big]
$$

- $T(\varnothing)$: the same template with the input removed or replaced by a domain premise such as the bare cue
- $\log P(v(y)\mid T(\varnothing))$: the label's prior: how much the model likes that word in this context regardless of input

The same idea, applied to the demonstrations in a few-shot prompt, is contextual calibration (see **Few-shot prompting**). Both cost one extra forward pass per label, and the prior can be cached because it does not depend on the input.

## Why zero-shot works at all

A base model is trained only to continue text (see **Language modeling**), so a zero-shot prompt works only if the pretraining data contained text shaped like "task description, then answer". For base models that is weak. GPT-3's zero-shot scores trailed its few-shot scores on most tasks, and the authors noted that zero-shot is sometimes hard even for humans to interpret without examples. **Instruction tuning** fixed most of this by training on many tasks phrased as instructions. **RLHF** then tuned models to follow instructions the way people want. Today's chat models are zero-shot learners by construction.

**Reference results**

- **GPT-3 175B, LAMBADA:** 76.2% zero-shot, 86.4% few-shot (Brown et al., 2020)
- **GPT-3 175B, TriviaQA:** 64.3% zero-shot, 68.0% one-shot, 71.2% few-shot, closed book
- **FLAN 137B (Wei et al., 2022):** instruction-tuned; zero-shot FLAN beat zero-shot GPT-3 175B on 20 of 25 datasets
- **InstructGPT (Ouyang et al., 2022):** labellers preferred outputs of the 1.3B instruction-following model over those of 175B GPT-3
- **Prompt format sensitivity:** Sclar et al. (2024) found accuracy spreads of up to 76 points across meaning-preserving format changes (spacing, separators, casing) for LLaMA-2-13B

## Try it: template, label words and readout

> Figure: A toy model classifies 40 reviews (20 positive) zero-shot. Label logits are ±c·s plus a per-word bias; the template sets c and how likely filler tokens are. Default (question template, great / terrible): free generation gets 22/40 and produces a label at all for only 23; label scoring gets 32/40 but says "great" for 28; PMI scoring gets 39/40. Review #15 ("Watchable, but forgettable.", negative): the greedy token is "The" (0.345); the label pair is 0.707 / 0.293 for great, while PMI gives great −0.145 and terrible +0.474. The instruction template with positive / negative gives 39, 39, 40 and 40/40 valid. The bare template with positive / negative produces no valid label at all (0/40), yet label scoring still gets 34/40 and PMI 37/40.

The three readouts fail in different ways. Free generation fails on **format**: the model may say something true in a form your parser rejects. Label scoring fixes the format but inherits the **prior** of the chosen words: "great" is a common, positive-leaning word, so it wins borderline cases. PMI subtracts that prior. A clear instruction with an explicit cue fixes most of the format problem by itself, which is why production prompts always say exactly what to output.

## Worked example: PMI flips a borderline call

The diagram's numbers. Natural logs; the prior comes from the same prompt with the review removed.

| Step | positive | negative | Decision |
| --- | --- | --- | --- |
| P(label \| full prompt), all tokens | 0.42 | 0.18 | — |
| Renormalise over the two labels | 0.42/0.60 = 0.700 | 0.18/0.60 = 0.300 | positive |
| Prior P(label \| no review) | 0.80 | 0.20 | — |
| PMI = ln(0.700/0.80), ln(0.300/0.20) | ln 0.875 = −0.134 | ln 1.5 = +0.405 | negative |
| Normalised ratio | 0.875/2.375 = 0.368 | 1.5/2.375 = 0.632 | negative |

The review raised "negative" from 0.20 to 0.30 and lowered "positive" from 0.80 to 0.70. Plain scoring sees only that 0.70 > 0.30. PMI sees which label the input pushed up.

## Writing zero-shot prompts

1. State the task and **the complete label set or output format** in the instruction: "Answer with one word: positive or negative."
2. End with an output cue (`Sentiment:`, `Answer:`) or, in chat, a format rule; ask for JSON when a program consumes the result (see **Structured output (JSON mode)**).
3. Separate the input from the instructions with delimiters (quotes, XML-style tags, triple backticks) so the model cannot confuse the two, and so injected text is easier to spot (see **Prompt injection**).
4. For classification with log-probability access, score the label words instead of parsing; add the content-free prior subtraction if one label dominates.
5. Use greedy decoding (temperature 0) for labels and extraction so runs are repeatable.
6. Test several paraphrases of the instruction on a labelled sample of 50–200 items before trusting one; if accuracy is still short, add examples (**Few-shot prompting**) or reasoning (**Chain-of-thought prompting**).

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("gpt2")
lm = AutoModelForCausalLM.from_pretrained("gpt2").eval()

@torch.no_grad()
def logprob(prompt: str, continuation: str) -> float:
    """Sum of log p(continuation tokens | prompt)."""
    p, c = tok(prompt).input_ids, tok(continuation).input_ids
    logp = lm(torch.tensor([p + c])).logits.log_softmax(-1)[0, len(p) - 1 : -1]
    return logp.gather(1, torch.tensor(c)[:, None]).sum().item()

TEMPLATE = ("Classify the sentiment of the review. Answer with one word: "
            "positive or negative.\nReview: {x}\nSentiment:")
LABELS = [" positive", " negative"]        # leading space: GPT-2 word tokens

def classify(x: str, pmi: bool = True) -> str:
    s = [logprob(TEMPLATE.format(x=x), y) for y in LABELS]
    if pmi:                                 # subtract the content-free prior
        s = [a - logprob(TEMPLATE.format(x=""), y) for a, y in zip(s, LABELS)]
    return LABELS[max(range(len(s)), key=s.__getitem__)].strip()

print(classify("Watchable, but forgettable."))
```

Zero-shot classification by label scoring with optional PMI, using Hugging Face transformers and GPT-2 (not run here: transformers and torch were not available in this environment)

> **One prompt is one sample:** Zero-shot accuracy depends on wording, label words, separators and even whitespace. A single prompt evaluated once tells you about that prompt, not about the model's ability. Report the spread over a few paraphrases, keep the winning template under version control (see **Prompt templates**), and re-test it when the model version changes: a prompt tuned for one model often loses accuracy on the next.

> **Zero-shot is a baseline, not a ceiling:** Zero-shot gives the cheapest working system and the number every other method must beat. If it is close, improve the instruction. If the model misunderstands the format or label boundaries, add a few examples. If it lacks knowledge, add retrieval. If it needs a narrow style at high volume, fine-tune. **Fine-tuning vs RAG vs prompting** lays out that decision.

**Strengths**

- No labelled data and the shortest prompt: lowest cost and latency of any prompting method.
- Changing the task is an edit to one sentence; no retraining.
- Label scoring guarantees a valid label and gives a probability you can threshold.

**Limitations**

- The model must infer the label boundaries and format; ambiguous instructions fail silently.
- Sensitive to wording and label-word priors; results can swing widely between paraphrases.
- Weak on unusual formats and specialised tasks the model rarely saw described in words.

**Question:** Label scoring returns "great" for 28 of 40 reviews when only 20 are positive. What does PMI scoring change?

- A. It uses a larger model
- B. It subtracts each label word's log-probability under the prompt with the input removed, cancelling the word's prior
- C. It samples several outputs and votes
- D. It adds examples of each label to the prompt

**Answer:** B. It subtracts each label word's log-probability under the prompt with the input removed, cancelling the word's prior — The excess "great" predictions come from the word's prior, not from the reviews. $\log P(y\mid T(x))-\log P(y\mid T(\varnothing))$ measures how much the input raised each label. In the toy at the default settings this takes accuracy from 32/40 to 39/40.

**Question:** A base model's zero-shot output often doesn't parse: it starts "The movie…" instead of giving a label. Which fix does not need log-probabilities?

- A. PMI scoring
- B. Contextual calibration
- C. An instruction that lists the allowed answers, plus an output cue such as "Sentiment:"
- D. Scoring the label words

**Answer:** C. An instruction that lists the allowed answers, plus an output cue such as "Sentiment:" — The other three all read token probabilities. A clear instruction and cue make the label the natural next token: in the toy, valid outputs go from 23/40 (question template) to 40/40 (instruction template).

**Key points**

- Zero-shot: instruction + input + output cue, no demonstrations, frozen weights (terminology from Brown et al., 2020).
- Classify by scoring the label words, $\arg\max_y \log P(v(y)\mid T(x))$: always a valid label, but biased towards frequent words.
- PMI scoring subtracts $\log P(v(y)\mid T(\varnothing))$, the label's prior with the input removed (Holtzman et al., 2021).
- Instruction tuning and RLHF are what made zero-shot reliable; base models do much better with examples.
- Prompt wording alone can move accuracy by tens of points: test paraphrases on a labelled sample and version the winning template.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
