# Few-shot prompting

> Few-shot prompting puts k input→output demonstrations before the query so a frozen model infers the task in context (Brown et al. 2020); accuracy swings with example choice, label balance and order, and contextual calibration (Zhao et al. 2021) removes much of that bias.

Source: https://www.mlfoundry.in/learn/few-shot-prompting  
Section: Generative AI › Prompt Engineering  
Level: core · 8 min read · updated 2026-10-21

**Few-shot prompting:** is placing $k$ solved examples (demonstrations) of a task in the prompt, followed by the new input, so that the model infers the task and the output format from them. The weights are not updated. Brown et al. ("Language Models are Few-Shot Learners", NeurIPS 2020) called this **in-context learning** and showed it at scale with GPT-3 175B. Typical $k$ is 1 to 10 in practice. The GPT-3 paper used 10 to 100, as many as fit in its 2,048-token window.

Few-shot sits between **Zero-shot prompting** and fine-tuning. A description alone often leaves the label boundaries, the output format or the style unclear. A handful of examples pins them down without collecting a training set or running a training job. The price is prompt length: every request carries the demonstrations, so they add input tokens and latency to every call. A stable demonstration prefix can be cached (see **Prompt caching**).

Demonstrations must share one exact format with the query. Use the same field names, separators and label spelling, and end with the same cue the examples use before their answers. With a chat API, each example is usually a user turn followed by an assistant turn, and the instruction goes in the system message.

> Figure: Top: three demonstrations (two positive, one negative, the last positive) and a test review. The demonstrations fix the label space, the input distribution and the format. Middle: the raw label probabilities, 0.62 / 0.38, are pulled towards "positive" by the majority label, recency and a common-token prior. Bottom: contextual calibration runs the same prompt with the input "N/A" (0.70 / 0.30), divides and renormalises: 0.886 and 1.267 become 0.411 / 0.589, so negative. Numbers are illustrative.

## What the model conditions on

$$
\hat y=\arg\max_{y\in\mathcal Y}\;P_\theta\big(y\mid I,\;x_1,y_1,\;\dots,\;x_k,y_k,\;x\big)
$$

- $I$: optional instruction (a few-shot prompt can have none)
- $(x_j,y_j)$: the j-th demonstration, in prompt order; order is part of the input
- $x$: the test input; the prediction is read after the shared output cue

What do the demonstrations actually teach? Min et al. ("Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?", EMNLP 2022) replaced the gold labels in demonstrations with random labels. Accuracy dropped only slightly on most classification and multiple-choice tasks. What mattered more was the **label space** (which answers exist), the **input distribution** (what inputs look like) and the **format**. Wei et al. (2023) found that the largest models do follow flipped labels when given enough of them, overriding their prior, while smaller models mostly ignore them. In practice: correct labels still help, but format and coverage come first.

## Bias, order and calibration

Zhao et al. ("Calibrate Before Use: Improving Few-Shot Performance of Language Models", ICML 2021) showed that GPT-3's few-shot accuracy can vary from near chance to near state of the art depending on the prompt format, which examples are chosen, and even their order. They traced most of the variance to three biases: **majority label** (the model favours the label that appears most in the demonstrations), **recency** (it favours the label of the last few demonstrations) and **common token** (it favours label words frequent in pretraining). Lu et al. ("Fantastically Ordered Prompts and Where to Find Them", ACL 2022) found the same order sensitivity across model sizes.

Their fix, **contextual calibration**, estimates the bias directly. Replace the test input with a content-free string such as "N/A", "[MASK]" or the empty string, keep the demonstrations, and read the label distribution $\hat p_{cf}$. A model with no bias would give a uniform distribution. Then rescale every real prediction:

$$
\hat q=\operatorname{normalize}\!\big(W\hat p\big),\qquad W=\operatorname{diag}(\hat p_{cf})^{-1},\qquad \hat y=\arg\max_y \hat q_y
$$

- $\hat p$: label probabilities for the real input, renormalised over the label set
- $\hat p_{cf}$: label probabilities for the content-free input under the same demonstrations (averaged over a few content-free strings)
- $W$: chosen so the content-free input maps to the uniform distribution; for two labels this subtracts the content-free log-odds

**From the papers**

- **GPT-3 175B (Brown et al., 2020):** LAMBADA 76.2% zero-shot → 86.4% few-shot; the few-shot gain over zero-shot grows with model size across the paper's 8 model sizes (125M–175B)
- **Contextual calibration (Zhao et al., 2021):** no training and no labelled data; raised GPT-3 and GPT-2 average accuracy by up to 30 points absolute on some tasks and reduced variance across prompt choices
- **Random labels (Min et al., 2022):** replacing gold labels with random ones cost only a few points across 12 models, including GPT-3
- **Retrieved examples (Liu et al., 2022):** choosing demonstrations by nearest-neighbour similarity to the query ("What Makes Good In-Context Examples for GPT-3?") beat random selection on several tasks

## Try it: the same examples in every order

> Figure: A toy in-context classifier, 40 test reviews (20 positive). Its "positive" logit adds a common-token bias (0.4), a majority-label term and a recency-weighted label term to the review's own evidence. Default (k = 4, two positive, positives last): 28/40 correct, "positive" predicted 32 times, the N/A probe gives p(pos) = 0.832; calibrated, 37/40. The six orderings of the same four labels range from 28 to 38 correct (mean 34.5). With negatives last: 35/40, p_cf = 0.310. With three positives placed last: 21/40, with 39 positive predictions. At k = 8 balanced, the 70 orderings span 24–38 (mean 33.8); calibration gives 37 for every one.

More shots raise mean accuracy only a little here, from 32 at $k=0$ to about 34 for balanced $k\geq2$ ($k=1$ and $k=3$ cannot be balanced and do worse). They do not narrow the spread, because every extra example adds room for imbalance and order effects. Calibration lifts the worst orderings to 37 and costs one point on the luckiest ones (38), whose bias happened to point the right way. In this toy the bias is purely additive, so calibration removes it exactly. In real models it removes only part of it.

## Worked example: the content-free probe

The toy's content-free logit d_cf = 0.4 + 1.5·m + 2.0·r for two orders of the same 2 + 2 labels. Recency weights halve per step back from the query.

| Quantity | − − + + (positives last) | + + − − (negatives last) |
| --- | --- | --- |
| Weights, first → last | 0.125, 0.25, 0.5, 1 | 0.125, 0.25, 0.5, 1 |
| Majority m = (n₊ − n₋)/k | 0 | 0 |
| Recency r = Σwℓ / Σw | (−0.125 − 0.25 + 0.5 + 1)/1.875 = 0.6 | −0.6 |
| d_cf = 0.4 + 2.0·r | 1.6 | −0.8 |
| p_cf(pos) = σ(d_cf) | 0.832 | 0.310 |
| Raw accuracy / calibrated | 28/40 / 37/40 | 35/40 / 37/40 |

Same examples, same labels, same count. Only the order differs, and the content-free probe swings from 0.832 to 0.310. Calibrating subtracts $d_{cf}$ from every test logit, so both orders give the same 37/40.

## Choosing and arranging examples

1. Cover the **label space**: include every label at least once, and keep counts roughly balanced unless you want the prior to follow the real class mix.
2. Match the **input distribution**: examples should look like production inputs in length, domain and difficulty; include an edge case the instruction alone gets wrong.
3. Keep one exact **format**: identical field names, separators and label spellings in every example and in the query cue.
4. Choose examples per query by **embedding similarity** when you have a labelled pool (see **Embedding models**), and dedupe near-identical ones.
5. Vary **order** on a dev set; if results swing, shuffle per request or calibrate, rather than trusting one lucky order.
6. Measure the gain over zero-shot on 100+ labelled items, and re-check when the model changes; instruction-tuned models often need fewer examples than base models.

```python
import numpy as np

def pick_demos(q_vec, pool_vecs, pool, k=4):
    """k nearest demonstrations by cosine, label-balanced, most similar placed last."""
    sims = pool_vecs @ q_vec / (np.linalg.norm(pool_vecs, axis=1) * np.linalg.norm(q_vec))
    by_label = {}
    for i in np.argsort(-sims):                       # most similar first
        by_label.setdefault(pool[i]["label"], []).append(i)
    per = k // len(by_label)
    chosen = [i for ids in by_label.values() for i in ids[:per]]
    return [pool[i] for i in sorted(chosen, key=lambda i: sims[i])]   # ascending similarity

def build_messages(demos, query):
    msgs = [{"role": "system", "content": "Classify the sentiment of each review. "
                                          "Reply with exactly one word: positive or negative."}]
    for d in demos:                                   # demonstrations as prior chat turns
        msgs += [{"role": "user", "content": d["text"]},
                 {"role": "assistant", "content": d["label"]}]
    return msgs + [{"role": "user", "content": query}]

pool = [{"text": t, "label": l} for t, l in [
    ("A gripping, beautiful film.", "positive"), ("Two hours I will never get back.", "negative"),
    ("The cast is wonderful.", "positive"), ("Dull and far too long.", "negative"),
    ("Funny from start to finish.", "positive"), ("The plot makes no sense.", "negative")]]
rng = np.random.default_rng(0)
pool_vecs = rng.normal(size=(6, 8))                   # stand-in for an embedding model
q_vec = pool_vecs[3] + 0.3 * rng.normal(size=8)       # a query close to "Dull and far too long."
demos = pick_demos(q_vec, pool_vecs, pool, k=4)
print([d["text"] for d in demos])
# ['Funny from start to finish.', 'The cast is wonderful.',
#  'Two hours I will never get back.', 'Dull and far too long.']
print(len(build_messages(demos, "Watchable, but forgettable.")))  # 1 + 2*4 + 1 = 10
```

Per-query example selection (k nearest by cosine, label-balanced, most similar last) and chat-format few-shot messages. Runs with numpy; the random vectors stand in for an embedding model.

> **Placing the closest example last adds recency bias:** Putting the most similar example right before the query helps when its label is usually the right one, which is why the code does it. It also makes that one label the most recent, so a misleading near-duplicate pulls the prediction its way. Check on a dev set whether "most similar last" beats a balanced shuffle for your task. Never let test items leak into the example pool, or the evaluation measures copying.

> **Few-shot for generation tasks:** For extraction, rewriting or code, examples mainly teach format and style. Two or three short, varied examples usually beat many long ones: the model copies surface features, so if every example is three sentences long, so is the answer. When the format must be exact, combine one example with **Structured output (JSON mode)** or **Constrained decoding** rather than adding more shots.

**Strengths**

- Specifies label boundaries, format and style that are hard to put into words.
- No training: a few labelled examples, changeable per request.
- Large gains over zero-shot for base models and unusual formats.

**Limitations**

- Every request pays for the demonstrations in tokens and latency.
- Sensitive to example choice, label balance and order; needs calibration or a dev set to trust.
- Gains shrink for strong instruction-tuned models and saturate after a few examples; fine-tuning wins at high volume.

**Question:** Four demonstrations, two of each label. In one order the model predicts "positive" for 32 of 40 test items (20 are positive); in another, for 15. What is the most likely cause?

- A. Majority-label bias
- B. Recency bias: the label of the last demonstrations pulls predictions
- C. The model has forgotten the instruction
- D. The test set changed

**Answer:** B. Recency bias: the label of the last demonstrations pulls predictions — Both orders have the same label counts, so the majority term is zero. Only the order differs, which is the recency bias of Zhao et al. (2021). In the toy, the content-free probe moves from 0.832 to 0.310 between the two orders.

**Question:** How does contextual calibration estimate the bias it removes?

- A. From a labelled validation set
- B. By fine-tuning a bias vector
- C. By scoring a content-free input such as "N/A" under the same demonstrations and dividing predictions by those probabilities
- D. By averaging over every ordering of the demonstrations

**Answer:** C. By scoring a content-free input such as "N/A" under the same demonstrations and dividing predictions by those probabilities — A content-free input carries no evidence, so an unbiased model would give a uniform distribution over labels. Whatever it gives instead is the bias. $W=\operatorname{diag}(\hat p_{cf})^{-1}$ maps it back to uniform and is applied to every test prediction. No labels and no training are needed.

**Key points**

- Few-shot = k demonstrations + query in one prompt, weights frozen; named in-context learning by Brown et al. (2020).
- Demonstrations mostly teach the label space, input distribution and format; random labels cost only a few points (Min et al., 2022).
- Majority-label, recency and common-token biases make accuracy depend on choice and order (Zhao et al., 2021; Lu et al., 2022).
- Contextual calibration: score "N/A", set $W=\operatorname{diag}(\hat p_{cf})^{-1}$, renormalise; no labels needed.
- Choose examples by similarity, balance labels, keep one format, check order on a dev set, and remember every shot costs tokens on every call.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
