Few-shot prompting

Few-shot prompting puts k input→output demonstrations before the query so a frozen model infers the task in context (Brown et al. 2020); accuracy swings with example choice, label balance and order, and contextual calibration (Zhao et al. 2021) removes much of that bias.

8 min read943 wordscore0 viewsupdated 2026-10-21

Definition

Few-shot prompting is placing kk solved examples (demonstrations) of a task in the prompt, followed by the new input, so that the model infers the task and the output format from them. The weights are not updated. Brown et al. ("Language Models are Few-Shot Learners", NeurIPS 2020) called this in-context learning and showed it at scale with GPT-3 175B. Typical kk is 1 to 10 in practice. The GPT-3 paper used 10 to 100, as many as fit in its 2,048-token window.

Few-shot sits between Zero-shot prompting and fine-tuning. A description alone often leaves the label boundaries, the output format or the style unclear. A handful of examples pins them down without collecting a training set or running a training job. The price is prompt length: every request carries the demonstrations, so they add input tokens and latency to every call. A stable demonstration prefix can be cached (see Prompt caching).

Demonstrations must share one exact format with the query. Use the same field names, separators and label spelling, and end with the same cue the examples use before their answers. With a chat API, each example is usually a user turn followed by an assistant turn, and the instruction goes in the system message.

few-shot prompt: k = 3 demonstrations, then the test input; weights stay frozenReview: A gripping, beautiful film.Sentiment: positiveReview: Two hours I will never get back.Sentiment: negativeReview: The cast is wonderful.Sentiment: positiveReview: Watchable, but forgettable.Sentiment: ▍the demonstrations fix:· the label space· the input distribution· the output formatrandom labels cost little(Min et al., 2022)raw next-token probabilities, label words onlypositive0.62negative0.38pulled towards "positive" by: majority label (2 of 3) · recency (last demo positive) · common tokencontextual calibration (Zhao et al., 2021): measure the bias, divide it out1 content-free probesame demos + "Review: N/A"p_cf = 0.70 / 0.302 real test inputsame demos + the reviewp = 0.62 / 0.383 q ∝ p ÷ p_cf0.62/0.70 = 0.8860.38/0.30 = 1.267→ 0.411 / 0.589: negativeprobabilities shown as positive / negative; numbers are illustrative, the procedure is the paper's
fig.Top: three demonstrations (two positive, one negative, the last positive) and a test review. The demonstrations fix the label space, the input distribution and the format. Middle: the raw label probabilities, 0.62 / 0.38, are pulled towards "positive" by the majority label, recency and a common-token prior. Bottom: contextual calibration runs the same prompt with the input "N/A" (0.70 / 0.30), divides and renormalises: 0.886 and 1.267 become 0.411 / 0.589, so negative. Numbers are illustrative.

What the model conditions on

y^=arg⁡max⁡y∈Y  Pθ(y∣I,  x1,y1,  …,  xk,yk,  x)\hat y=\arg\max_{y\in\mathcal Y}\;P_\theta\big(y\mid I,\;x_1,y_1,\;\dots,\;x_k,y_k,\;x\big)
II
optional instruction (a few-shot prompt can have none)
(xj,yj)(x_j,y_j)
the j-th demonstration, in prompt order; order is part of the input
xx
the test input; the prediction is read after the shared output cue

What do the demonstrations actually teach? Min et al. ("Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?", EMNLP 2022) replaced the gold labels in demonstrations with random labels. Accuracy dropped only slightly on most classification and multiple-choice tasks. What mattered more was the label space (which answers exist), the input distribution (what inputs look like) and the format. Wei et al. (2023) found that the largest models do follow flipped labels when given enough of them, overriding their prior, while smaller models mostly ignore them. In practice: correct labels still help, but format and coverage come first.

Bias, order and calibration

Zhao et al. ("Calibrate Before Use: Improving Few-Shot Performance of Language Models", ICML 2021) showed that GPT-3's few-shot accuracy can vary from near chance to near state of the art depending on the prompt format, which examples are chosen, and even their order. They traced most of the variance to three biases: majority label (the model favours the label that appears most in the demonstrations), recency (it favours the label of the last few demonstrations) and common token (it favours label words frequent in pretraining). Lu et al. ("Fantastically Ordered Prompts and Where to Find Them", ACL 2022) found the same order sensitivity across model sizes.

Their fix, contextual calibration, estimates the bias directly. Replace the test input with a content-free string such as "N/A", "[MASK]" or the empty string, keep the demonstrations, and read the label distribution p^cf\hat p_{cf}. A model with no bias would give a uniform distribution. Then rescale every real prediction:

q^=normalize⁡ ⁣(Wp^),W=diag⁡(p^cf)−1,y^=arg⁡max⁡yq^y\hat q=\operatorname{normalize}\!\big(W\hat p\big),\qquad W=\operatorname{diag}(\hat p_{cf})^{-1},\qquad \hat y=\arg\max_y \hat q_y
p^\hat p
label probabilities for the real input, renormalised over the label set
p^cf\hat p_{cf}
label probabilities for the content-free input under the same demonstrations (averaged over a few content-free strings)
WW
chosen so the content-free input maps to the uniform distribution; for two labels this subtracts the content-free log-odds

From the papers

GPT-3 175B (Brown et al., 2020)
LAMBADA 76.2% zero-shot → 86.4% few-shot; the few-shot gain over zero-shot grows with model size across the paper's 8 model sizes (125M–175B)
Contextual calibration (Zhao et al., 2021)
no training and no labelled data; raised GPT-3 and GPT-2 average accuracy by up to 30 points absolute on some tasks and reduced variance across prompt choices
Random labels (Min et al., 2022)
replacing gold labels with random ones cost only a few points across 12 models, including GPT-3
Retrieved examples (Liu et al., 2022)
choosing demonstrations by nearest-neighbour similarity to the query ("What Makes Good In-Context Examples for GPT-3?") beat random selection on several tasks

Try it: the same examples in every order

prompt: 4 demonstrations (2 positive, 2 negative) + test review#1→ neg0.13#2→ neg0.25#3→ pos0.50#4→ pos1.00test review→ ?"N/A" (probe)p(pos) 0.832← recency weights (×0.5/step)this order: accuracy 28/40, predicts "positive" for 32/40 (truth 20)calibrated (subtract the N/A logit, i.e. divide by p_cf): 37/40all 6 orderings of these labels: 28–38 correct, mean 34.515202530354021111correct of 40 · bar = # orderings · ┆ calibrated, any ordervs k (n₊ = ⌊k/2⌋, all orders)203040012345678— mean, band min–max · - - calibratedtoy logit for "positive" on a review with sentiment s:d = c(k)·s + ε + 0.4 + 1.5·m + 2.0·r0.4 common-token bias · m = (n₊ − n₋)/k majorityr = recency-weighted mean label · c(k) = 2 − e^(−k/2)calibration: d − d("N/A") removes the last three termshere exactly; in a real model only partly
order
fig.A toy in-context classifier, 40 test reviews (20 positive). Its "positive" logit adds a common-token bias (0.4), a majority-label term and a recency-weighted label term to the review's own evidence. Default (k = 4, two positive, positives last): 28/40 correct, "positive" predicted 32 times, the N/A probe gives p(pos) = 0.832; calibrated, 37/40. The six orderings of the same four labels range from 28 to 38 correct (mean 34.5). With negatives last: 35/40, p_cf = 0.310. With three positives placed last: 21/40, with 39 positive predictions. At k = 8 balanced, the 70 orderings span 24–38 (mean 33.8); calibration gives 37 for every one.

More shots raise mean accuracy only a little here, from 32 at k=0k=0 to about 34 for balanced k≥2k\geq2 (k=1k=1 and k=3k=3 cannot be balanced and do worse). They do not narrow the spread, because every extra example adds room for imbalance and order effects. Calibration lifts the worst orderings to 37 and costs one point on the luckiest ones (38), whose bias happened to point the right way. In this toy the bias is purely additive, so calibration removes it exactly. In real models it removes only part of it.

Worked example: the content-free probe

Quantity− − + + (positives last)+ + − − (negatives last)
Weights, first → last0.125, 0.25, 0.5, 10.125, 0.25, 0.5, 1
Majority m = (n₊ − n₋)/k00
Recency r = Σwℓ / Σw(−0.125 − 0.25 + 0.5 + 1)/1.875 = 0.6−0.6
d_cf = 0.4 + 2.0·r1.6−0.8
p_cf(pos) = σ(d_cf)0.8320.310
Raw accuracy / calibrated28/40 / 37/4035/40 / 37/40
The toy's content-free logit d_cf = 0.4 + 1.5·m + 2.0·r for two orders of the same 2 + 2 labels. Recency weights halve per step back from the query.

Same examples, same labels, same count. Only the order differs, and the content-free probe swings from 0.832 to 0.310. Calibrating subtracts dcfd_{cf} from every test logit, so both orders give the same 37/40.

Choosing and arranging examples

  1. 1Cover the label space: include every label at least once, and keep counts roughly balanced unless you want the prior to follow the real class mix.
  2. 2Match the input distribution: examples should look like production inputs in length, domain and difficulty; include an edge case the instruction alone gets wrong.
  3. 3Keep one exact format: identical field names, separators and label spellings in every example and in the query cue.
  4. 4Choose examples per query by embedding similarity when you have a labelled pool (see Embedding models), and dedupe near-identical ones.
  5. 5Vary order on a dev set; if results swing, shuffle per request or calibrate, rather than trusting one lucky order.
  6. 6Measure the gain over zero-shot on 100+ labelled items, and re-check when the model changes; instruction-tuned models often need fewer examples than base models.
python
1import numpy as np
2
3def pick_demos(q_vec, pool_vecs, pool, k=4):
4 """k nearest demonstrations by cosine, label-balanced, most similar placed last."""
5 sims = pool_vecs @ q_vec / (np.linalg.norm(pool_vecs, axis=1) * np.linalg.norm(q_vec))
6 by_label = {}
7 for i in np.argsort(-sims): # most similar first
8 by_label.setdefault(pool[i]["label"], []).append(i)
9 per = k // len(by_label)
10 chosen = [i for ids in by_label.values() for i in ids[:per]]
11 return [pool[i] for i in sorted(chosen, key=lambda i: sims[i])] # ascending similarity
12
13def build_messages(demos, query):
14 msgs = [{"role": "system", "content": "Classify the sentiment of each review. "
15 "Reply with exactly one word: positive or negative."}]
16 for d in demos: # demonstrations as prior chat turns
17 msgs += [{"role": "user", "content": d["text"]},
18 {"role": "assistant", "content": d["label"]}]
19 return msgs + [{"role": "user", "content": query}]
20
21pool = [{"text": t, "label": l} for t, l in [
22 ("A gripping, beautiful film.", "positive"), ("Two hours I will never get back.", "negative"),
23 ("The cast is wonderful.", "positive"), ("Dull and far too long.", "negative"),
24 ("Funny from start to finish.", "positive"), ("The plot makes no sense.", "negative")]]
25rng = np.random.default_rng(0)
26pool_vecs = rng.normal(size=(6, 8)) # stand-in for an embedding model
27q_vec = pool_vecs[3] + 0.3 * rng.normal(size=8) # a query close to "Dull and far too long."
28demos = pick_demos(q_vec, pool_vecs, pool, k=4)
29print([d["text"] for d in demos])
30# ['Funny from start to finish.', 'The cast is wonderful.',
31# 'Two hours I will never get back.', 'Dull and far too long.']
32print(len(build_messages(demos, "Watchable, but forgettable."))) # 1 + 2*4 + 1 = 10
Per-query example selection (k nearest by cosine, label-balanced, most similar last) and chat-format few-shot messages. Runs with numpy; the random vectors stand in for an embedding model.

Use when

  • +Specifies label boundaries, format and style that are hard to put into words.
  • +No training: a few labelled examples, changeable per request.
  • +Large gains over zero-shot for base models and unusual formats.

Avoid when

  • −Every request pays for the demonstrations in tokens and latency.
  • −Sensitive to example choice, label balance and order; needs calibration or a dev set to trust.
  • −Gains shrink for strong instruction-tuned models and saturate after a few examples; fine-tuning wins at high volume.

Check yourself

Four demonstrations, two of each label. In one order the model predicts "positive" for 32 of 40 test items (20 are positive); in another, for 15. What is the most likely cause?

Check yourself

How does contextual calibration estimate the bias it removes?

Remember

  • ✓Few-shot = k demonstrations + query in one prompt, weights frozen; named in-context learning by Brown et al. (2020).
  • ✓Demonstrations mostly teach the label space, input distribution and format; random labels cost only a few points (Min et al., 2022).
  • ✓Majority-label, recency and common-token biases make accuracy depend on choice and order (Zhao et al., 2021; Lu et al., 2022).
  • ✓Contextual calibration: score "N/A", set W=diag⁡(p^cf)−1W=\operatorname{diag}(\hat p_{cf})^{-1}, renormalise; no labels needed.
  • ✓Choose examples by similarity, balance labels, keep one format, check order on a dev set, and remember every shot costs tokens on every call.
← back to Generative AI roadmap