Definition
Few-shot prompting is placing solved examples (demonstrations) of a task in the prompt, followed by the new input, so that the model infers the task and the output format from them. The weights are not updated. Brown et al. ("Language Models are Few-Shot Learners", NeurIPS 2020) called this in-context learning and showed it at scale with GPT-3 175B. Typical is 1 to 10 in practice. The GPT-3 paper used 10 to 100, as many as fit in its 2,048-token window.
Few-shot sits between Zero-shot prompting and fine-tuning. A description alone often leaves the label boundaries, the output format or the style unclear. A handful of examples pins them down without collecting a training set or running a training job. The price is prompt length: every request carries the demonstrations, so they add input tokens and latency to every call. A stable demonstration prefix can be cached (see Prompt caching).
Demonstrations must share one exact format with the query. Use the same field names, separators and label spelling, and end with the same cue the examples use before their answers. With a chat API, each example is usually a user turn followed by an assistant turn, and the instruction goes in the system message.
What the model conditions on
- optional instruction (a few-shot prompt can have none)
- the j-th demonstration, in prompt order; order is part of the input
- the test input; the prediction is read after the shared output cue
What do the demonstrations actually teach? Min et al. ("Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?", EMNLP 2022) replaced the gold labels in demonstrations with random labels. Accuracy dropped only slightly on most classification and multiple-choice tasks. What mattered more was the label space (which answers exist), the input distribution (what inputs look like) and the format. Wei et al. (2023) found that the largest models do follow flipped labels when given enough of them, overriding their prior, while smaller models mostly ignore them. In practice: correct labels still help, but format and coverage come first.
Bias, order and calibration
Zhao et al. ("Calibrate Before Use: Improving Few-Shot Performance of Language Models", ICML 2021) showed that GPT-3's few-shot accuracy can vary from near chance to near state of the art depending on the prompt format, which examples are chosen, and even their order. They traced most of the variance to three biases: majority label (the model favours the label that appears most in the demonstrations), recency (it favours the label of the last few demonstrations) and common token (it favours label words frequent in pretraining). Lu et al. ("Fantastically Ordered Prompts and Where to Find Them", ACL 2022) found the same order sensitivity across model sizes.
Their fix, contextual calibration, estimates the bias directly. Replace the test input with a content-free string such as "N/A", "[MASK]" or the empty string, keep the demonstrations, and read the label distribution . A model with no bias would give a uniform distribution. Then rescale every real prediction:
- label probabilities for the real input, renormalised over the label set
- label probabilities for the content-free input under the same demonstrations (averaged over a few content-free strings)
- chosen so the content-free input maps to the uniform distribution; for two labels this subtracts the content-free log-odds
From the papers
- GPT-3 175B (Brown et al., 2020)
- LAMBADA 76.2% zero-shot → 86.4% few-shot; the few-shot gain over zero-shot grows with model size across the paper's 8 model sizes (125M–175B)
- Contextual calibration (Zhao et al., 2021)
- no training and no labelled data; raised GPT-3 and GPT-2 average accuracy by up to 30 points absolute on some tasks and reduced variance across prompt choices
- Random labels (Min et al., 2022)
- replacing gold labels with random ones cost only a few points across 12 models, including GPT-3
- Retrieved examples (Liu et al., 2022)
- choosing demonstrations by nearest-neighbour similarity to the query ("What Makes Good In-Context Examples for GPT-3?") beat random selection on several tasks
Try it: the same examples in every order
More shots raise mean accuracy only a little here, from 32 at to about 34 for balanced ( and cannot be balanced and do worse). They do not narrow the spread, because every extra example adds room for imbalance and order effects. Calibration lifts the worst orderings to 37 and costs one point on the luckiest ones (38), whose bias happened to point the right way. In this toy the bias is purely additive, so calibration removes it exactly. In real models it removes only part of it.
Worked example: the content-free probe
| Quantity | − − + + (positives last) | + + − − (negatives last) |
|---|---|---|
| Weights, first → last | 0.125, 0.25, 0.5, 1 | 0.125, 0.25, 0.5, 1 |
| Majority m = (n₊ − n₋)/k | 0 | 0 |
| Recency r = Σwℓ / Σw | (−0.125 − 0.25 + 0.5 + 1)/1.875 = 0.6 | −0.6 |
| d_cf = 0.4 + 2.0·r | 1.6 | −0.8 |
| p_cf(pos) = σ(d_cf) | 0.832 | 0.310 |
| Raw accuracy / calibrated | 28/40 / 37/40 | 35/40 / 37/40 |
Same examples, same labels, same count. Only the order differs, and the content-free probe swings from 0.832 to 0.310. Calibrating subtracts from every test logit, so both orders give the same 37/40.
Choosing and arranging examples
- 1Cover the label space: include every label at least once, and keep counts roughly balanced unless you want the prior to follow the real class mix.
- 2Match the input distribution: examples should look like production inputs in length, domain and difficulty; include an edge case the instruction alone gets wrong.
- 3Keep one exact format: identical field names, separators and label spellings in every example and in the query cue.
- 4Choose examples per query by embedding similarity when you have a labelled pool (see Embedding models), and dedupe near-identical ones.
- 5Vary order on a dev set; if results swing, shuffle per request or calibrate, rather than trusting one lucky order.
- 6Measure the gain over zero-shot on 100+ labelled items, and re-check when the model changes; instruction-tuned models often need fewer examples than base models.
1import numpy as np23def pick_demos(q_vec, pool_vecs, pool, k=4):4 """k nearest demonstrations by cosine, label-balanced, most similar placed last."""5 sims = pool_vecs @ q_vec / (np.linalg.norm(pool_vecs, axis=1) * np.linalg.norm(q_vec))6 by_label = {}7 for i in np.argsort(-sims): # most similar first8 by_label.setdefault(pool[i]["label"], []).append(i)9 per = k // len(by_label)10 chosen = [i for ids in by_label.values() for i in ids[:per]]11 return [pool[i] for i in sorted(chosen, key=lambda i: sims[i])] # ascending similarity1213def build_messages(demos, query):14 msgs = [{"role": "system", "content": "Classify the sentiment of each review. "15 "Reply with exactly one word: positive or negative."}]16 for d in demos: # demonstrations as prior chat turns17 msgs += [{"role": "user", "content": d["text"]},18 {"role": "assistant", "content": d["label"]}]19 return msgs + [{"role": "user", "content": query}]2021pool = [{"text": t, "label": l} for t, l in [22 ("A gripping, beautiful film.", "positive"), ("Two hours I will never get back.", "negative"),23 ("The cast is wonderful.", "positive"), ("Dull and far too long.", "negative"),24 ("Funny from start to finish.", "positive"), ("The plot makes no sense.", "negative")]]25rng = np.random.default_rng(0)26pool_vecs = rng.normal(size=(6, 8)) # stand-in for an embedding model27q_vec = pool_vecs[3] + 0.3 * rng.normal(size=8) # a query close to "Dull and far too long."28demos = pick_demos(q_vec, pool_vecs, pool, k=4)29print([d["text"] for d in demos])30# ['Funny from start to finish.', 'The cast is wonderful.',31# 'Two hours I will never get back.', 'Dull and far too long.']32print(len(build_messages(demos, "Watchable, but forgettable."))) # 1 + 2*4 + 1 = 10Use when
- +Specifies label boundaries, format and style that are hard to put into words.
- +No training: a few labelled examples, changeable per request.
- +Large gains over zero-shot for base models and unusual formats.
Avoid when
- −Every request pays for the demonstrations in tokens and latency.
- −Sensitive to example choice, label balance and order; needs calibration or a dev set to trust.
- −Gains shrink for strong instruction-tuned models and saturate after a few examples; fine-tuning wins at high volume.
Check yourself
Four demonstrations, two of each label. In one order the model predicts "positive" for 32 of 40 test items (20 are positive); in another, for 15. What is the most likely cause?
Check yourself
How does contextual calibration estimate the bias it removes?
Remember
- ✓Few-shot = k demonstrations + query in one prompt, weights frozen; named in-context learning by Brown et al. (2020).
- ✓Demonstrations mostly teach the label space, input distribution and format; random labels cost only a few points (Min et al., 2022).
- ✓Majority-label, recency and common-token biases make accuracy depend on choice and order (Zhao et al., 2021; Lu et al., 2022).
- ✓Contextual calibration: score "N/A", set , renormalise; no labels needed.
- ✓Choose examples by similarity, balance labels, keep one format, check order on a dev set, and remember every shot costs tokens on every call.