Definition
Self-consistency is a decoding strategy for chain-of-thought prompting: sample independent reasoning paths from the same prompt at a non-zero temperature, extract each path's final answer, and return the answer that occurs most often. It was introduced by Wang, Wei, Schuurmans, Le, Chi, Narang, Chowdhery and Zhou in "Self-Consistency Improves Chain of Thought Reasoning in Language Models" (arXiv 2022, ICLR 2023).
Chain-of-thought prompting asks the model to write out its reasoning before the answer. With greedy decoding, the model commits to one path. One wrong step early on, such as forgetting a quantity, carries through to a wrong answer, and nothing in the output marks it as wrong.
The observation behind self-consistency: a hard problem has many valid derivations that reach the same answer, while mistakes tend to scatter over different wrong answers. Sampling several paths and voting over the answers lets the correct answer win even when no single path is reliable.
It changes only the decoding step. There is no training, no verifier model and no change to the prompt. In a system it sits between the model API and the answer parser: request completions, parse each one, then aggregate.
The rule
- the -th sampled reasoning path (discarded after parsing)
- the final answer parsed from path , normalised (e.g. "18.00" and "18" are equal)
- number of samples; the paper used 40
- the plurality answer; ties need an explicit rule
The paper frames this as marginalising out the reasoning. The quantity you want is , the probability of an answer summed over every path that reaches it. Greedy decoding instead finds one high-probability path, which need not lead to the most probable answer. The vote is a Monte Carlo estimate of the marginal.
When the vote helps
Take the simplest case: each path is correct with probability , and every wrong path gives the same wrong answer. With odd, the vote is correct when more than half the paths are:
- accuracy rises towards 1 as grows (Condorcet's jury theorem)
- accuracy falls towards 0: the vote amplifies a systematic error
Real wrong answers are spread out, so the true condition is weaker: the correct answer only needs to be the most likely single answer (the plurality), not a majority. A question where 45% of paths are right and the rest split over five wrong answers still improves with . A question with a popular misconception does not, because the wrong answer has the most votes.
Paper results (Wang et al., 2022)
- GSM8K, PaLM-540B
- 56.5% with chain-of-thought and greedy decoding → 74.4% with self-consistency over 40 paths
- Other gains, PaLM-540B
- SVAMP +11.0, AQuA +12.2, StrategyQA +6.4, ARC-challenge +3.9 accuracy points over greedy chain-of-thought
- Protocol
- 40 sampled outputs per question; reported results averaged over 10 runs
- Aggregation
- the plain unweighted vote did about as well as weighting each path by its length-normalised probability; the unnormalised weighted sum was worse
- Sampling
- gains held across temperature, top-k and nucleus settings; accuracy rose fastest over the first few paths and kept rising more slowly up to 40
Worked example
| Per-path accuracy p | n = 1 | n = 5 | n = 11 | n = 21 | n = 41 |
|---|---|---|---|---|---|
| 0.60 | 0.600 | 0.683 | 0.753 | 0.826 | 0.903 |
| 0.55 | 0.550 | 0.593 | 0.633 | 0.679 | 0.741 |
| 0.40 | 0.400 | 0.317 | 0.247 | 0.174 | 0.097 |
For and : . Five times the tokens buys 8.3 points here; 41 times the tokens buys 30.3. A benchmark is a mix of such questions, so its accuracy is the average of rising curves (questions the model usually gets right) and falling ones (questions it usually gets wrong).
1import re2from collections import Counter3from openai import OpenAI45client = OpenAI()6ANS = re.compile(r"answer is\s*\$?(-?[\d,]*\.?\d+)", re.I)78def normalise(s: str) -> str:9 return str(float(s.replace(",", ""))).removesuffix(".0") # "18.00" -> "18"1011def self_consistent(question: str, cot_prompt: str, n: int = 10) -> tuple[str, float]:12 resp = client.chat.completions.create(13 model="gpt-4o-mini",14 messages=[{"role": "system", "content": cot_prompt},15 {"role": "user", "content": question}],16 n=n, temperature=0.7, max_tokens=400,17 )18 answers = [normalise(m.group(1)) for c in resp.choices19 if (m := ANS.search(c.message.content or ""))]20 if not answers:21 raise ValueError("no parsable answer in any sample")22 best, votes = Counter(answers).most_common(1)[0]23 return best, votes / n # vote share doubles as a confidence scoreUse when
- +No training, no extra model: one parameter (n) on top of an existing chain-of-thought prompt.
- +Large gains on tasks with a short, checkable answer: arithmetic, multiple choice, classification.
- +The n samples run in parallel, so latency grows far less than cost.
Avoid when
- −Cost scales linearly with n in output tokens; the prompt can be shared through Prompt caching or the API's n parameter.
- −Needs answers that can be compared for equality; free-form text needs a different aggregator.
- −Cannot fix a systematic error: if most paths share a misconception, more samples make the wrong answer more certain.
Check yourself
A model gets a question right on 30% of sampled paths; 42% of paths give one particular wrong answer and the rest are scattered. What happens as n grows from 1 to 40?
Check yourself
Why does self-consistency sample at temperature > 0 rather than run beam search to get n paths?
Remember
- ✓Sample chain-of-thought paths at , parse each final answer, return the plurality: .
- ✓It approximates the answer's marginal probability, summing over all reasoning paths that reach it.
- ✓PaLM-540B, 40 paths: GSM8K 56.5% → 74.4%; accuracy rises fastest over the first few samples.
- ✓Helps when the correct answer is the plurality; amplifies errors when a wrong answer is.
- ✓Costs n generations; vote share doubles as a confidence score. For search over partial reasoning, see Tree of thoughts.