# Self-consistency

> Sample many chain-of-thought paths at T > 0 and return the most frequent final answer; with 40 paths it lifted PaLM-540B on GSM8K from 56.5% to 74.4% (Wang et al., 2022), at n times the cost.

Source: https://www.mlfoundry.in/learn/self-consistency  
Section: Generative AI › Prompt Engineering  
Level: core · 7 min read · updated 2026-10-21

**Self-consistency:** is a decoding strategy for chain-of-thought prompting: sample $n$ independent reasoning paths from the same prompt at a non-zero temperature, extract each path's final answer, and return the answer that occurs most often. It was introduced by Wang, Wei, Schuurmans, Le, Chi, Narang, Chowdhery and Zhou in "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (arXiv 2022, ICLR 2023).

**Chain-of-thought prompting** asks the model to write out its reasoning before the answer. With greedy decoding, the model commits to one path. One wrong step early on, such as forgetting a quantity, carries through to a wrong answer, and nothing in the output marks it as wrong.

The observation behind self-consistency: a hard problem has many valid derivations that reach the same answer, while mistakes tend to scatter over different wrong answers. Sampling several paths and voting over the answers lets the correct answer win even when no single path is reliable.

It changes only the decoding step. There is no training, no verifier model and no change to the prompt. In a system it sits between the model API and the answer parser: request $n$ completions, parse each one, then aggregate.

## The rule

$$
(r_i,a_i)\sim p_\theta(\cdot\mid \text{prompt},q),\;\; i=1..n,\qquad \hat a=\arg\max_{a}\sum_{i=1}^{n}\mathbb 1[a_i=a]
$$

- $r_i$: the $i$-th sampled reasoning path (discarded after parsing)
- $a_i$: the final answer parsed from path $i$, normalised (e.g. "18.00" and "18" are equal)
- $n$: number of samples; the paper used 40
- $\hat a$: the plurality answer; ties need an explicit rule

The paper frames this as marginalising out the reasoning. The quantity you want is $P(a\mid q)=\sum_r P(r,a\mid q)$, the probability of an answer summed over every path that reaches it. Greedy decoding instead finds one high-probability path, which need not lead to the most probable answer. The vote is a Monte Carlo estimate of the marginal.

> Figure: The GSM8K-style question from the paper's running example (16 − 3 − 4 = 9 eggs, 9 × $2 = $18). The greedy path forgets the muffins and returns $26. Five sampled paths (illustrative) end in $18, $26, $18, $9 and $18. The vote over final answers returns $18 with 3 of 5 votes. The reasoning text itself is never compared.

## When the vote helps

Take the simplest case: each path is correct with probability $p$, and every wrong path gives the same wrong answer. With $n$ odd, the vote is correct when more than half the paths are:

$$
P(\hat a\text{ correct})=\sum_{k>n/2}\binom{n}{k}p^{k}(1-p)^{n-k}
$$

- $p>0.5$: accuracy rises towards 1 as $n$ grows (Condorcet's jury theorem)
- $p<0.5$: accuracy falls towards 0: the vote amplifies a systematic error

Real wrong answers are spread out, so the true condition is weaker: the correct answer only needs to be the most likely single answer (the plurality), not a majority. A question where 45% of paths are right and the rest split over five wrong answers still improves with $n$. A question with a popular misconception does not, because the wrong answer has the most votes.

**Paper results (Wang et al., 2022)**

- **GSM8K, PaLM-540B:** 56.5% with chain-of-thought and greedy decoding → 74.4% with self-consistency over 40 paths
- **Other gains, PaLM-540B:** SVAMP +11.0, AQuA +12.2, StrategyQA +6.4, ARC-challenge +3.9 accuracy points over greedy chain-of-thought
- **Protocol:** 40 sampled outputs per question; reported results averaged over 10 runs
- **Aggregation:** the plain unweighted vote did about as well as weighting each path by its length-normalised probability; the unnormalised weighted sum was worse
- **Sampling:** gains held across temperature, top-k and nucleus settings; accuracy rose fastest over the first few paths and kept rising more slowly up to 40

> Figure: Exact plurality-vote accuracy for one question, with uniform tie-breaking. Each path is correct ($18) with probability p; a share s of the wrong mass goes to $26 and the rest is split over four other answers. At the defaults (p = 0.45, s = 0.5, so $26 gets 0.275) accuracy is 45.0% for one path, 61.2% at n = 5, 72.3% at n = 10 and 90.6% at n = 40. Set s = 1 (all wrong paths say $26, 0.55) and the vote gets worse with n: 42.5% at n = 3, 26.4% at n = 40. At p = 0.35, s = 0.5 the correct answer leads by only 0.025 and n = 40 reaches just 57.6%.

## Worked example

Two-answer case: P(vote correct) = Σ_{k>n/2} C(n,k) pᵏ(1−p)ⁿ⁻ᵏ

| Per-path accuracy p | n = 1 | n = 5 | n = 11 | n = 21 | n = 41 |
| --- | --- | --- | --- | --- | --- |
| 0.60 | 0.600 | 0.683 | 0.753 | 0.826 | 0.903 |
| 0.55 | 0.550 | 0.593 | 0.633 | 0.679 | 0.741 |
| 0.40 | 0.400 | 0.317 | 0.247 | 0.174 | 0.097 |

For $p=0.6$ and $n=5$: $\binom53 0.6^3 0.4^2+\binom54 0.6^4 0.4+0.6^5=0.3456+0.2592+0.0778=0.6826$. Five times the tokens buys 8.3 points here; 41 times the tokens buys 30.3. A benchmark is a mix of such questions, so its accuracy is the average of rising curves (questions the model usually gets right) and falling ones (questions it usually gets wrong).

```python
import re
from collections import Counter
from openai import OpenAI

client = OpenAI()
ANS = re.compile(r"answer is\s*\$?(-?[\d,]*\.?\d+)", re.I)

def normalise(s: str) -> str:
    return str(float(s.replace(",", ""))).removesuffix(".0")   # "18.00" -> "18"

def self_consistent(question: str, cot_prompt: str, n: int = 10) -> tuple[str, float]:
    resp = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "system", "content": cot_prompt},
                  {"role": "user", "content": question}],
        n=n, temperature=0.7, max_tokens=400,
    )
    answers = [normalise(m.group(1)) for c in resp.choices
               if (m := ANS.search(c.message.content or ""))]
    if not answers:
        raise ValueError("no parsable answer in any sample")
    best, votes = Counter(answers).most_common(1)[0]
    return best, votes / n          # vote share doubles as a confidence score
```

Self-consistency with an OpenAI-compatible chat API (n completions in one request). The API call was not run here; the parse-and-vote part is plain Python.

> **The vote is only as good as the parser:** Votes are compared by exact string after normalisation. "18 dollars", "18" and "18.0" must map to one key, and paths with no parsable answer must be dropped, not counted as a vote for "None". For free-form outputs (summaries, code) there is no equality test. Universal self-consistency (Chen et al., 2023) instead asks the model to pick the most consistent of the sampled responses.

> **Agreement is a confidence signal:** The winning vote share is a cheap, well-behaved uncertainty estimate: in the paper, accuracy was higher on questions where the paths agreed more. Use it to route low-agreement questions to a stronger model or a human. Sampling can also stop early: once the leader is ahead by more votes than there are samples left, the result cannot change.

**Strengths**

- No training, no extra model: one parameter (n) on top of an existing chain-of-thought prompt.
- Large gains on tasks with a short, checkable answer: arithmetic, multiple choice, classification.
- The n samples run in parallel, so latency grows far less than cost.

**Limitations**

- Cost scales linearly with n in output tokens; the prompt can be shared through **Prompt caching** or the API's n parameter.
- Needs answers that can be compared for equality; free-form text needs a different aggregator.
- Cannot fix a systematic error: if most paths share a misconception, more samples make the wrong answer more certain.

**Question:** A model gets a question right on 30% of sampled paths; 42% of paths give one particular wrong answer and the rest are scattered. What happens as n grows from 1 to 40?

- A. Accuracy rises towards 100%
- B. Accuracy stays at 30%
- C. Accuracy falls, because the wrong answer is the plurality
- D. Accuracy rises only if temperature is raised

**Answer:** C. Accuracy falls, because the wrong answer is the plurality — The vote converges to the most probable single answer. Here that is the wrong one (0.42 > 0.30), so accuracy drops: in the viz's answer model it is 30.0% at n = 1 and 18.4% at n = 40. Self-consistency needs the correct answer to be the plurality.

**Question:** Why does self-consistency sample at temperature > 0 rather than run beam search to get n paths?

- A. Beam search cannot produce chain-of-thought text
- B. Beam outputs are near-duplicates that share early mistakes, so their votes are not independent
- C. Sampling is always cheaper per token
- D. Beam search needs a reward model

**Answer:** B. Beam outputs are near-duplicates that share early mistakes, so their votes are not independent — The vote helps only if errors are roughly independent across paths. Beams share prefixes and differ by a few tokens, so one early slip appears in most of them. The paper found self-consistency with sampling beat beam-search decoding at the same number of outputs.

**Key points**

- Sample $n$ chain-of-thought paths at $T>0$, parse each final answer, return the plurality: $\hat a=\arg\max_a\#\{i:a_i=a\}$.
- It approximates the answer's marginal probability, summing over all reasoning paths that reach it.
- PaLM-540B, 40 paths: GSM8K 56.5% → 74.4%; accuracy rises fastest over the first few samples.
- Helps when the correct answer is the plurality; amplifies errors when a wrong answer is.
- Costs n generations; vote share doubles as a confidence score. For search over partial reasoning, see **Tree of thoughts**.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
