Chain-of-thought prompting

Chain-of-thought prompting (Wei et al., 2022) makes a model write intermediate steps before its answer; with 8 worked exemplars PaLM 540B rose from 17.9% to 56.9% on GSM8K, at the cost of many more output tokens.

8 min read752 wordscore0 viewsupdated 2026-10-21

Definition

Chain-of-thought (CoT) prompting is a prompting technique in which the model is led to generate a series of intermediate reasoning steps in natural language before it states the final answer. Wei et al. introduced it in "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (NeurIPS 2022) using few-shot exemplars whose answers spell out the reasoning. Kojima et al. ("Large Language Models are Zero-Shot Reasoners", NeurIPS 2022) showed that the single trigger "Let's think step by step" gets much of the effect with no exemplars.

The problem is multi-step reasoning: arithmetic word problems, symbolic manipulation, questions that chain several facts. In Few-shot prompting the exemplars show only question and answer, so the model must produce the answer token straight away. Every intermediate quantity has to be computed inside a single forward pass per output token. A transformer has a fixed number of layers, so the amount of serial computation available before the first answer token is bounded.

CoT moves that computation into the output. Each written step becomes context for the next one, so the model reads its own intermediate results instead of holding them implicitly. In a product, CoT sits in the prompt (exemplars or an instruction), and an answer parser sits after generation. The parser pulls the final answer out of the reasoning text.

Two ways to elicit it

few-shot chain of thought: the exemplar shows the reasoning, not just the answerstandard promptingmodel inputQ: Roger has 5 tennis balls. He buys 2 morecans of tennis balls. Each can has 3 tennisballs. How many tennis balls does he have now?A: The answer is 11.Q: The cafeteria had 23 apples. If they used20 to make lunch and bought 6 more, how manyapples do they have?model outputA: The answer is 27.✗ wrong (truth 9)chain-of-thought promptingmodel inputQ: Roger has 5 tennis balls. He buys 2 morecans of tennis balls. Each can has 3 tennisballs. How many tennis balls does he have now?A: Roger started with 5 balls. 2 cans of 3tennis balls each is 6 tennis balls. 5 + 6 = 11.The answer is 11.Q: The cafeteria had 23 apples. If they used20 to make lunch and bought 6 more, how manyapples do they have?model outputA: The cafeteria had 23 apples originally.They used 20 to make lunch. So they had23 - 20 = 3. They bought 6 more apples, sothey have 3 + 6 = 9. The answer is 9.✓ correctzero-shot chain of thought: no exemplars, a trigger phrase and two callsQ: [question]A: Let's thinkstep by step.23 - 20 = 3.3 + 6 = 9.(reasoning)[Q] [A: Let's think…] [reasoning]Therefore, the answer(arabic numerals) is9stage 1 promptgeneratedstage 2 prompt (answer extraction)answertop: exemplar and outputs from Wei et al. (2022), Fig. 1 · bottom: Kojima et al. (2022) two-stage template
fig.Top: the exemplar and outputs of Wei et al. (2022), Figure 1. The same model, given an exemplar that shows only "The answer is 11", answers the cafeteria question with 27. Given an exemplar that writes out 2 × 3 = 6 and 5 + 6 = 11, it writes 23 − 20 = 3, 3 + 6 = 9 and answers 9. Bottom: zero-shot CoT (Kojima et al., 2022) uses two calls, one to generate the reasoning after the trigger phrase and one to extract the answer in a fixed format.

Variants used in practice

  1. 1Few-shot CoT. Put kk exemplars (Wei et al. used 8 for the maths benchmarks) of the form question → reasoning → "The answer is X." before the new question. The exemplars fix both the reasoning style and the answer format.
  2. 2Zero-shot CoT. Append "Let's think step by step." to the answer prefix, generate, then send a second prompt such as "Therefore, the answer (arabic numerals) is" to extract a clean answer.
  3. 3Instruction CoT. In chat models an instruction like "Reason step by step, then give the final answer on a line starting with Answer:" plays the same role and fixes the parse target.
  4. 4Structured decomposition. Least-to-most prompting (Zhou et al., 2022) first asks for subproblems, then solves them in order. Program-aided variants (PAL, Gao et al., 2022; Program of Thoughts, Chen et al., 2022) have the model write Python and let an interpreter do the arithmetic.

Why it helps, and when it does not

One way to read CoT is as a latent variable. The model's answer distribution marginalises over reasoning paths rr:

p(a∣q)=∑rp(r∣q) p(a∣q,r)p(a\mid q)=\sum_{r} p(r\mid q)\,p(a\mid q,r)
qq
the question plus the prompt (exemplars or trigger)
rr
a reasoning path, i.e. the generated chain of steps
aa
the final answer extracted after the chain

A single greedy CoT run samples one path. Self-consistency samples many and takes a majority vote over aa, which approximates the sum. Tree of thoughts searches over partial paths instead of sampling whole ones.

A simple error model shows why CoT pays off on long problems. Suppose the model performs each written step correctly with probability pp. Answering directly, it must compose all nn steps internally and succeeds with a lower per-step rate a<pa<p:

PCoT(n)≈p n,Pdirect(n)≈a n,PCoTPdirect=(pa)nP_{\text{CoT}}(n)\approx p^{\,n},\qquad P_{\text{direct}}(n)\approx a^{\,n},\qquad \frac{P_{\text{CoT}}}{P_{\text{direct}}}=\left(\frac{p}{a}\right)^{n}
nn
number of dependent reasoning steps
pp
probability that one written step is right
aa
effective per-step success when the steps stay implicit

The ratio grows exponentially with nn when p>ap>a. If the model's written steps are worse than its intuition (p<ap<a), CoT hurts. Wei et al. saw exactly this. Below roughly 100B parameters, models wrote fluent but illogical chains, and CoT gave no gain or lowered accuracy. This is one of the standard examples in Emergent abilities.

Reported results (checkable in the papers)

Few-shot CoT, GSM8K
PaLM 540B: 17.9% with standard prompting, 56.9% with 8 CoT exemplars (Wei et al., 2022)
Zero-shot CoT
text-davinci-002 with "Let's think step by step": MultiArith 17.7% → 78.7%, GSM8K 10.4% → 40.7% (Kojima et al., 2022)
Scale
gains appeared only at about 10¹¹ parameters (LaMDA, GPT-3, PaLM families); smaller models were flat or worse
Faithfulness
Turpin et al. (2023) biased few-shot prompts (e.g. the right option always "(A)"); models followed the bias while their written CoT never mentioned it

Simulate it

problem #1, n = 4 steps · large model: step accuracy p = 0.96, direct a = 0.72 per stepQ: Start with 16. Add 9. Add 4. Multiply by 2. Add 6.standard prompt (answer only)A: The answer is 73. ✗ (truth 64)chain-of-thought prompt (reasoning, then answer)16 + 9 = 25.25 + 4 = 29.29 × 2 = 58.58 + 6 = 64.The answer is 64. ✓accuracy over 200 problems vs n0%50%100%12345678● CoT · ■ direct · dashed: pⁿ and aⁿat n = 4: CoT 169/200 (84.5%, pⁿ = 84.9%) · direct 42/200 (21.0%, aⁿ = 26.9%)CoT output ≈ 16.8 tokens (chars / 4) vs about 5 for the bare answer · gain +63.5 pointstoy simulation: p and a are made-up per-size rates, not measured from real models
model size
fig.200 seeded arithmetic problems per length n, solved by a toy model in two modes. CoT steps are right with probability p and a slip propagates; the direct answer is right with probability aⁿ. Default (large model, p = 0.96, a = 0.72, n = 4): CoT 169/200 (84.5%, pⁿ = 84.9%), direct 42/200 (21.0%, aⁿ = 26.9%). The CoT trace costs about 16.8 tokens against about 5 for a bare answer. Small model (p = 0.55, a = 0.65): at n = 4, CoT 19/200 against direct 32/200, so CoT hurts. Large model at n = 8: 143 against 11.

Worked example: accuracy against cost

QuantityDirect answerChain of thought
Accuracy at n = 10.72¹ = 72.0%0.96¹ = 96.0%
Accuracy at n = 40.72⁴ = 26.9%0.96⁴ = 84.9%
Accuracy at n = 80.72⁸ = 7.2%0.96⁸ = 72.1%
Output tokens5100 (typical short chain)
Latency0.3 + 5 × 0.025 = 0.425 s0.3 + 100 × 0.025 = 2.8 s
The error model with p = 0.96, a = 0.72, and latency = TTFT + output tokens × time per output token, with TTFT = 0.3 s and 25 ms per token.

At n=4n=4 CoT turns a 26.9% task into an 84.9% one, for about 6.6× the latency and 20× the output tokens. At n=1n=1 the gain is 24 points for the same price. For single-step lookups or classification, that price is often not worth paying. See Cost and latency optimization.

python
1import re
2
3EXEMPLAR = ("Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has "
4 "3 tennis balls. How many tennis balls does he have now?\n"
5 "A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. "
6 "5 + 6 = 11. The answer is 11.\n\n")
7
8def few_shot_cot(question):
9 return EXEMPLAR + f"Q: {question}\nA:"
10
11def zero_shot_cot(question, complete):
12 """Kojima et al. (2022): reasoning extraction, then answer extraction."""
13 stage1 = f"Q: {question}\nA: Let's think step by step."
14 reasoning = complete(stage1)
15 stage2 = stage1 + reasoning + "\nTherefore, the answer (arabic numerals) is"
16 return reasoning, complete(stage2)
17
18def extract_answer(text):
19 """Prefer 'The answer is N'; fall back to the last number in the text."""
20 m = re.search(r"answer is\s*\$?(-?[\d,]*\.?\d+)", text)
21 nums = [m.group(1)] if m else re.findall(r"-?[\d,]*\.?\d+", text)
22 return float(nums[-1].replace(",", "")) if nums else None
23
24out = ("The cafeteria had 23 apples originally. They used 20 to make lunch. So they had "
25 "23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9.")
26print(extract_answer(out)) # 9.0
27print(extract_answer("so 1,250 + 30 = 1,280")) # 1280.0
Few-shot and zero-shot CoT prompts with an answer extractor. `complete` stands for any text-completion call; the extractor was run on the sample outputs shown (prints 9.0 and 1280.0).

Use when

  • +Large gains on multi-step maths, symbolic and multi-hop tasks for capable models
  • +No training: exemplars or one instruction
  • +Intermediate steps make many errors easy to spot and test
  • +Composes with self-consistency, tool use (ReAct prompting) and verifiers

Avoid when

  • −10–100× more output tokens: higher latency and cost
  • −Little or negative gain for small models and single-step tasks
  • −Chains can be fluent and wrong; one slip propagates to the answer
  • −Needs answer extraction; free-form chains break naive parsers (pair with Structured output (JSON mode))

Check yourself

Under the error model, a model writes each step correctly with p = 0.9 but composes steps internally with a = 0.95. What does CoT do on a 6-step problem?

Check yourself

Why does zero-shot CoT (Kojima et al.) use a second prompt after the reasoning?

Remember

  • ✓CoT makes the model write intermediate steps, so each step can read the previous results from context.
  • ✓Few-shot CoT uses exemplars with worked reasoning. Zero-shot CoT uses "Let's think step by step" plus an answer-extraction prompt.
  • ✓Gains grow with the number of dependent steps, roughly as (p/a)n(p/a)^n, and appear only in sufficiently capable models.
  • ✓The cost is output tokens and latency. Skip CoT for single-step tasks.
  • ✓The chain is not a faithful explanation. Verify answers instead of trusting the reasoning text.
← back to Generative AI roadmap