Self-consistency

Sample many chain-of-thought paths at T > 0 and return the most frequent final answer; with 40 paths it lifted PaLM-540B on GSM8K from 56.5% to 74.4% (Wang et al., 2022), at n times the cost.

7 min read600 wordscore0 viewsupdated 2026-10-21

Definition

Self-consistency is a decoding strategy for chain-of-thought prompting: sample nn independent reasoning paths from the same prompt at a non-zero temperature, extract each path's final answer, and return the answer that occurs most often. It was introduced by Wang, Wei, Schuurmans, Le, Chi, Narang, Chowdhery and Zhou in "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (arXiv 2022, ICLR 2023).

Chain-of-thought prompting asks the model to write out its reasoning before the answer. With greedy decoding, the model commits to one path. One wrong step early on, such as forgetting a quantity, carries through to a wrong answer, and nothing in the output marks it as wrong.

The observation behind self-consistency: a hard problem has many valid derivations that reach the same answer, while mistakes tend to scatter over different wrong answers. Sampling several paths and voting over the answers lets the correct answer win even when no single path is reliable.

It changes only the decoding step. There is no training, no verifier model and no change to the prompt. In a system it sits between the model API and the answer parser: request nn completions, parse each one, then aggregate.

The rule

(ri,ai)∼pθ(⋅∣prompt,q),    i=1..n,a^=arg⁡max⁡a∑i=1n1[ai=a](r_i,a_i)\sim p_\theta(\cdot\mid \text{prompt},q),\;\; i=1..n,\qquad \hat a=\arg\max_{a}\sum_{i=1}^{n}\mathbb 1[a_i=a]
rir_i
the ii-th sampled reasoning path (discarded after parsing)
aia_i
the final answer parsed from path ii, normalised (e.g. "18.00" and "18" are equal)
nn
number of samples; the paper used 40
a^\hat a
the plurality answer; ties need an explicit rule

The paper frames this as marginalising out the reasoning. The quantity you want is P(a∣q)=∑rP(r,a∣q)P(a\mid q)=\sum_r P(r,a\mid q), the probability of an answer summed over every path that reaches it. Greedy decoding instead finds one high-probability path, which need not lead to the most probable answer. The vote is a Monte Carlo estimate of the marginal.

prompt: few-shot chain-of-thought exemplars, thenQ: Janet's ducks lay 16 eggs a day. She eats 3 and bakes muffins with 4.She sells the rest at $2 per egg. How much does she make a day? A:greedy decode (T = 0)16 − 3 = 13 eggs left;13 × $2 = $26$26 ✗an early slipcannot be undonesample n = 5 reasoning paths (T = 0.7)16 − 3 − 4 = 9 eggs left; 9 × $2 = $18$18she eats 3: 16 − 3 = 13; 13 × $2 = $26$263 + 4 = 7 used; 16 − 7 = 9; 9 × 2 = 18$1816 − 3 − 4 = 9 eggs are sold$9(16 − 7) eggs × $2 per egg = $18$18voteover answers$18 ×3$26 ×1$9 ×1$18 ✓â = argmax_a #{i : aᵢ = a} (reasoning is discarded; only final answers vote)cost: n full generations instead of one · needs a parsable final answer · ties need a rule
fig.The GSM8K-style question from the paper's running example (16 − 3 − 4 = 9 eggs, 9 × $2 = $18). The greedy path forgets the muffins and returns $26. Five sampled paths (illustrative) end in $18, $26, $18, $9 and $18. The vote over final answers returns $18 with 3 of 5 votes. The reasoning text itself is never compared.

When the vote helps

Take the simplest case: each path is correct with probability pp, and every wrong path gives the same wrong answer. With nn odd, the vote is correct when more than half the paths are:

P(a^ correct)=∑k>n/2(nk)pk(1−p)n−kP(\hat a\text{ correct})=\sum_{k>n/2}\binom{n}{k}p^{k}(1-p)^{n-k}
p>0.5p>0.5
accuracy rises towards 1 as nn grows (Condorcet's jury theorem)
p<0.5p<0.5
accuracy falls towards 0: the vote amplifies a systematic error

Real wrong answers are spread out, so the true condition is weaker: the correct answer only needs to be the most likely single answer (the plurality), not a majority. A question where 45% of paths are right and the rest split over five wrong answers still improves with nn. A question with a popular misconception does not, because the wrong answer has the most votes.

Paper results (Wang et al., 2022)

GSM8K, PaLM-540B
56.5% with chain-of-thought and greedy decoding → 74.4% with self-consistency over 40 paths
Other gains, PaLM-540B
SVAMP +11.0, AQuA +12.2, StrategyQA +6.4, ARC-challenge +3.9 accuracy points over greedy chain-of-thought
Protocol
40 sampled outputs per question; reported results averaged over 10 runs
Aggregation
the plain unweighted vote did about as well as weighting each path by its length-normalised probability; the unnormalised weighted sum was worse
Sampling
gains held across temperature, top-k and nucleus settings; accuracy rose fastest over the first few paths and kept rising more slowly up to 40
answer = most frequent final answer among n sampled reasoning pathsP(path ends in answer)$18 ✓0.450$260.275$90.069$140.069$220.069$300.069one draw of n = 10 paths (seed 1)votes: $26×4 $18×3 $30×3vote → $26 (wrong)exact P(vote correct) vs n00.51110203040sampled paths none pathvote accuracy at n = 1072.3%one sampled path45.0%change / token cost+27.3 pt / 10×correct answer is the plurality: vote → 1top wrong answer: 0.275
sampled vote
fig.Exact plurality-vote accuracy for one question, with uniform tie-breaking. Each path is correct ($18) with probability p; a share s of the wrong mass goes to $26 and the rest is split over four other answers. At the defaults (p = 0.45, s = 0.5, so $26 gets 0.275) accuracy is 45.0% for one path, 61.2% at n = 5, 72.3% at n = 10 and 90.6% at n = 40. Set s = 1 (all wrong paths say $26, 0.55) and the vote gets worse with n: 42.5% at n = 3, 26.4% at n = 40. At p = 0.35, s = 0.5 the correct answer leads by only 0.025 and n = 40 reaches just 57.6%.

Worked example

Per-path accuracy pn = 1n = 5n = 11n = 21n = 41
0.600.6000.6830.7530.8260.903
0.550.5500.5930.6330.6790.741
0.400.4000.3170.2470.1740.097
Two-answer case: P(vote correct) = Σ_{k>n/2} C(n,k) pᵏ(1−p)ⁿ⁻ᵏ

For p=0.6p=0.6 and n=5n=5: (53)0.630.42+(54)0.640.4+0.65=0.3456+0.2592+0.0778=0.6826\binom53 0.6^3 0.4^2+\binom54 0.6^4 0.4+0.6^5=0.3456+0.2592+0.0778=0.6826. Five times the tokens buys 8.3 points here; 41 times the tokens buys 30.3. A benchmark is a mix of such questions, so its accuracy is the average of rising curves (questions the model usually gets right) and falling ones (questions it usually gets wrong).

python
1import re
2from collections import Counter
3from openai import OpenAI
4
5client = OpenAI()
6ANS = re.compile(r"answer is\s*\$?(-?[\d,]*\.?\d+)", re.I)
7
8def normalise(s: str) -> str:
9 return str(float(s.replace(",", ""))).removesuffix(".0") # "18.00" -> "18"
10
11def self_consistent(question: str, cot_prompt: str, n: int = 10) -> tuple[str, float]:
12 resp = client.chat.completions.create(
13 model="gpt-4o-mini",
14 messages=[{"role": "system", "content": cot_prompt},
15 {"role": "user", "content": question}],
16 n=n, temperature=0.7, max_tokens=400,
17 )
18 answers = [normalise(m.group(1)) for c in resp.choices
19 if (m := ANS.search(c.message.content or ""))]
20 if not answers:
21 raise ValueError("no parsable answer in any sample")
22 best, votes = Counter(answers).most_common(1)[0]
23 return best, votes / n # vote share doubles as a confidence score
Self-consistency with an OpenAI-compatible chat API (n completions in one request). The API call was not run here; the parse-and-vote part is plain Python.

Use when

  • +No training, no extra model: one parameter (n) on top of an existing chain-of-thought prompt.
  • +Large gains on tasks with a short, checkable answer: arithmetic, multiple choice, classification.
  • +The n samples run in parallel, so latency grows far less than cost.

Avoid when

  • −Cost scales linearly with n in output tokens; the prompt can be shared through Prompt caching or the API's n parameter.
  • −Needs answers that can be compared for equality; free-form text needs a different aggregator.
  • −Cannot fix a systematic error: if most paths share a misconception, more samples make the wrong answer more certain.

Check yourself

A model gets a question right on 30% of sampled paths; 42% of paths give one particular wrong answer and the rest are scattered. What happens as n grows from 1 to 40?

Check yourself

Why does self-consistency sample at temperature > 0 rather than run beam search to get n paths?

Remember

  • ✓Sample nn chain-of-thought paths at T>0T>0, parse each final answer, return the plurality: a^=arg⁡max⁡a#{i:ai=a}\hat a=\arg\max_a\#\{i:a_i=a\}.
  • ✓It approximates the answer's marginal probability, summing over all reasoning paths that reach it.
  • ✓PaLM-540B, 40 paths: GSM8K 56.5% → 74.4%; accuracy rises fastest over the first few samples.
  • ✓Helps when the correct answer is the plurality; amplifies errors when a wrong answer is.
  • ✓Costs n generations; vote share doubles as a confidence score. For search over partial reasoning, see Tree of thoughts.
← back to Generative AI roadmap