Definition
Zero-shot prompting is asking a pretrained language model to do a task from a natural-language description alone, with no worked examples in the prompt and no change to the weights. Radford et al. ("Language Models are Unsupervised Multitask Learners", 2019, GPT-2) showed zero-shot task transfer from pretraining alone. Brown et al. ("Language Models are Few-Shot Learners", 2020, GPT-3) fixed the terms: zero-shot gives only an instruction, one-shot adds one demonstration, few-shot adds several.
It is the default way people use a chat assistant, and the first baseline for any new task. It is also the cheapest: the prompt is short, there is nothing to collect or label, and changing the task means editing a sentence. The cost is control. The model must infer the task, the label set and the output format from words alone. When any of these is ambiguous, accuracy drops, and the output may not parse.
A zero-shot prompt has three parts. The instruction states the task and the allowed outputs. The input is the item to process. The output cue is the text right before the answer, such as Sentiment:, A: or TL;DR:, and it tells the model what kind of token comes next. GPT-2 summarised articles zero-shot just by appending TL;DR:. In a chat model the instruction usually goes in the system message (see System prompts and role prompting), and the cue becomes an explicit format rule.
Reading the answer
For open-ended tasks (summaries, rewrites, answers) you decode text and use it. For classification there is a more reliable option: never let the model choose freely. Score each allowed label instead. A verbalizer maps each class to a label string, such as positive → " positive". The prediction is the class whose string is most likely after the prompt:
- the prompt template filled with input x (instruction, input, cue)
- the token sequence of the label for class y; one token in the simple case
- the closed label set; nothing outside it can be predicted
This removes format failures, because the answer is always a valid label. It does not remove bias. Some label words are simply more frequent, so the model prefers them whatever the input says. Labels of several tokens also compete unequally: a summed log-probability penalises longer strings, which is why some evaluations divide by the token count.
Holtzman et al. ("Surface Form Competition: Why the Highest Probability Answer Isn't Always Right", EMNLP 2021) proposed scoring by how much the input raises each label's probability. Their domain-conditional PMI divides by the label's probability under a prompt that keeps the task wording but drops the input:
- the same template with the input removed or replaced by a domain premise such as the bare cue
- the label's prior: how much the model likes that word in this context regardless of input
The same idea, applied to the demonstrations in a few-shot prompt, is contextual calibration (see Few-shot prompting). Both cost one extra forward pass per label, and the prior can be cached because it does not depend on the input.
Why zero-shot works at all
A base model is trained only to continue text (see Language modeling), so a zero-shot prompt works only if the pretraining data contained text shaped like "task description, then answer". For base models that is weak. GPT-3's zero-shot scores trailed its few-shot scores on most tasks, and the authors noted that zero-shot is sometimes hard even for humans to interpret without examples. Instruction tuning fixed most of this by training on many tasks phrased as instructions. RLHF then tuned models to follow instructions the way people want. Today's chat models are zero-shot learners by construction.
Reference results
- GPT-3 175B, LAMBADA
- 76.2% zero-shot, 86.4% few-shot (Brown et al., 2020)
- GPT-3 175B, TriviaQA
- 64.3% zero-shot, 68.0% one-shot, 71.2% few-shot, closed book
- FLAN 137B (Wei et al., 2022)
- instruction-tuned; zero-shot FLAN beat zero-shot GPT-3 175B on 20 of 25 datasets
- InstructGPT (Ouyang et al., 2022)
- labellers preferred outputs of the 1.3B instruction-following model over those of 175B GPT-3
- Prompt format sensitivity
- Sclar et al. (2024) found accuracy spreads of up to 76 points across meaning-preserving format changes (spacing, separators, casing) for LLaMA-2-13B
Try it: template, label words and readout
The three readouts fail in different ways. Free generation fails on format: the model may say something true in a form your parser rejects. Label scoring fixes the format but inherits the prior of the chosen words: "great" is a common, positive-leaning word, so it wins borderline cases. PMI subtracts that prior. A clear instruction with an explicit cue fixes most of the format problem by itself, which is why production prompts always say exactly what to output.
Worked example: PMI flips a borderline call
| Step | positive | negative | Decision |
|---|---|---|---|
| P(label | full prompt), all tokens | 0.42 | 0.18 | — |
| Renormalise over the two labels | 0.42/0.60 = 0.700 | 0.18/0.60 = 0.300 | positive |
| Prior P(label | no review) | 0.80 | 0.20 | — |
| PMI = ln(0.700/0.80), ln(0.300/0.20) | ln 0.875 = −0.134 | ln 1.5 = +0.405 | negative |
| Normalised ratio | 0.875/2.375 = 0.368 | 1.5/2.375 = 0.632 | negative |
The review raised "negative" from 0.20 to 0.30 and lowered "positive" from 0.80 to 0.70. Plain scoring sees only that 0.70 > 0.30. PMI sees which label the input pushed up.
Writing zero-shot prompts
- 1State the task and the complete label set or output format in the instruction: "Answer with one word: positive or negative."
- 2End with an output cue (
Sentiment:,Answer:) or, in chat, a format rule; ask for JSON when a program consumes the result (see Structured output (JSON mode)). - 3Separate the input from the instructions with delimiters (quotes, XML-style tags, triple backticks) so the model cannot confuse the two, and so injected text is easier to spot (see Prompt injection).
- 4For classification with log-probability access, score the label words instead of parsing; add the content-free prior subtraction if one label dominates.
- 5Use greedy decoding (temperature 0) for labels and extraction so runs are repeatable.
- 6Test several paraphrases of the instruction on a labelled sample of 50–200 items before trusting one; if accuracy is still short, add examples (Few-shot prompting) or reasoning (Chain-of-thought prompting).
1import torch2from transformers import AutoTokenizer, AutoModelForCausalLM34tok = AutoTokenizer.from_pretrained("gpt2")5lm = AutoModelForCausalLM.from_pretrained("gpt2").eval()67@torch.no_grad()8def logprob(prompt: str, continuation: str) -> float:9 """Sum of log p(continuation tokens | prompt)."""10 p, c = tok(prompt).input_ids, tok(continuation).input_ids11 logp = lm(torch.tensor([p + c])).logits.log_softmax(-1)[0, len(p) - 1 : -1]12 return logp.gather(1, torch.tensor(c)[:, None]).sum().item()1314TEMPLATE = ("Classify the sentiment of the review. Answer with one word: "15 "positive or negative.\nReview: {x}\nSentiment:")16LABELS = [" positive", " negative"] # leading space: GPT-2 word tokens1718def classify(x: str, pmi: bool = True) -> str:19 s = [logprob(TEMPLATE.format(x=x), y) for y in LABELS]20 if pmi: # subtract the content-free prior21 s = [a - logprob(TEMPLATE.format(x=""), y) for a, y in zip(s, LABELS)]22 return LABELS[max(range(len(s)), key=s.__getitem__)].strip()2324print(classify("Watchable, but forgettable."))Use when
- +No labelled data and the shortest prompt: lowest cost and latency of any prompting method.
- +Changing the task is an edit to one sentence; no retraining.
- +Label scoring guarantees a valid label and gives a probability you can threshold.
Avoid when
- −The model must infer the label boundaries and format; ambiguous instructions fail silently.
- −Sensitive to wording and label-word priors; results can swing widely between paraphrases.
- −Weak on unusual formats and specialised tasks the model rarely saw described in words.
Check yourself
Label scoring returns "great" for 28 of 40 reviews when only 20 are positive. What does PMI scoring change?
Check yourself
A base model's zero-shot output often doesn't parse: it starts "The movie…" instead of giving a label. Which fix does not need log-probabilities?
Remember
- ✓Zero-shot: instruction + input + output cue, no demonstrations, frozen weights (terminology from Brown et al., 2020).
- ✓Classify by scoring the label words, : always a valid label, but biased towards frequent words.
- ✓PMI scoring subtracts , the label's prior with the input removed (Holtzman et al., 2021).
- ✓Instruction tuning and RLHF are what made zero-shot reliable; base models do much better with examples.
- ✓Prompt wording alone can move accuracy by tens of points: test paraphrases on a labelled sample and version the winning template.