Definition
Chain-of-thought (CoT) prompting is a prompting technique in which the model is led to generate a series of intermediate reasoning steps in natural language before it states the final answer. Wei et al. introduced it in "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (NeurIPS 2022) using few-shot exemplars whose answers spell out the reasoning. Kojima et al. ("Large Language Models are Zero-Shot Reasoners", NeurIPS 2022) showed that the single trigger "Let's think step by step" gets much of the effect with no exemplars.
The problem is multi-step reasoning: arithmetic word problems, symbolic manipulation, questions that chain several facts. In Few-shot prompting the exemplars show only question and answer, so the model must produce the answer token straight away. Every intermediate quantity has to be computed inside a single forward pass per output token. A transformer has a fixed number of layers, so the amount of serial computation available before the first answer token is bounded.
CoT moves that computation into the output. Each written step becomes context for the next one, so the model reads its own intermediate results instead of holding them implicitly. In a product, CoT sits in the prompt (exemplars or an instruction), and an answer parser sits after generation. The parser pulls the final answer out of the reasoning text.
Two ways to elicit it
Variants used in practice
- 1Few-shot CoT. Put exemplars (Wei et al. used 8 for the maths benchmarks) of the form question → reasoning → "The answer is X." before the new question. The exemplars fix both the reasoning style and the answer format.
- 2Zero-shot CoT. Append "Let's think step by step." to the answer prefix, generate, then send a second prompt such as "Therefore, the answer (arabic numerals) is" to extract a clean answer.
- 3Instruction CoT. In chat models an instruction like "Reason step by step, then give the final answer on a line starting with Answer:" plays the same role and fixes the parse target.
- 4Structured decomposition. Least-to-most prompting (Zhou et al., 2022) first asks for subproblems, then solves them in order. Program-aided variants (PAL, Gao et al., 2022; Program of Thoughts, Chen et al., 2022) have the model write Python and let an interpreter do the arithmetic.
Why it helps, and when it does not
One way to read CoT is as a latent variable. The model's answer distribution marginalises over reasoning paths :
- the question plus the prompt (exemplars or trigger)
- a reasoning path, i.e. the generated chain of steps
- the final answer extracted after the chain
A single greedy CoT run samples one path. Self-consistency samples many and takes a majority vote over , which approximates the sum. Tree of thoughts searches over partial paths instead of sampling whole ones.
A simple error model shows why CoT pays off on long problems. Suppose the model performs each written step correctly with probability . Answering directly, it must compose all steps internally and succeeds with a lower per-step rate :
- number of dependent reasoning steps
- probability that one written step is right
- effective per-step success when the steps stay implicit
The ratio grows exponentially with when . If the model's written steps are worse than its intuition (), CoT hurts. Wei et al. saw exactly this. Below roughly 100B parameters, models wrote fluent but illogical chains, and CoT gave no gain or lowered accuracy. This is one of the standard examples in Emergent abilities.
Reported results (checkable in the papers)
- Few-shot CoT, GSM8K
- PaLM 540B: 17.9% with standard prompting, 56.9% with 8 CoT exemplars (Wei et al., 2022)
- Zero-shot CoT
- text-davinci-002 with "Let's think step by step": MultiArith 17.7% → 78.7%, GSM8K 10.4% → 40.7% (Kojima et al., 2022)
- Scale
- gains appeared only at about 10¹¹ parameters (LaMDA, GPT-3, PaLM families); smaller models were flat or worse
- Faithfulness
- Turpin et al. (2023) biased few-shot prompts (e.g. the right option always "(A)"); models followed the bias while their written CoT never mentioned it
Simulate it
Worked example: accuracy against cost
| Quantity | Direct answer | Chain of thought |
|---|---|---|
| Accuracy at n = 1 | 0.72¹ = 72.0% | 0.96¹ = 96.0% |
| Accuracy at n = 4 | 0.72⁴ = 26.9% | 0.96⁴ = 84.9% |
| Accuracy at n = 8 | 0.72⁸ = 7.2% | 0.96⁸ = 72.1% |
| Output tokens | 5 | 100 (typical short chain) |
| Latency | 0.3 + 5 × 0.025 = 0.425 s | 0.3 + 100 × 0.025 = 2.8 s |
At CoT turns a 26.9% task into an 84.9% one, for about 6.6× the latency and 20× the output tokens. At the gain is 24 points for the same price. For single-step lookups or classification, that price is often not worth paying. See Cost and latency optimization.
1import re23EXEMPLAR = ("Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has "4 "3 tennis balls. How many tennis balls does he have now?\n"5 "A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. "6 "5 + 6 = 11. The answer is 11.\n\n")78def few_shot_cot(question):9 return EXEMPLAR + f"Q: {question}\nA:"1011def zero_shot_cot(question, complete):12 """Kojima et al. (2022): reasoning extraction, then answer extraction."""13 stage1 = f"Q: {question}\nA: Let's think step by step."14 reasoning = complete(stage1)15 stage2 = stage1 + reasoning + "\nTherefore, the answer (arabic numerals) is"16 return reasoning, complete(stage2)1718def extract_answer(text):19 """Prefer 'The answer is N'; fall back to the last number in the text."""20 m = re.search(r"answer is\s*\$?(-?[\d,]*\.?\d+)", text)21 nums = [m.group(1)] if m else re.findall(r"-?[\d,]*\.?\d+", text)22 return float(nums[-1].replace(",", "")) if nums else None2324out = ("The cafeteria had 23 apples originally. They used 20 to make lunch. So they had "25 "23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9.")26print(extract_answer(out)) # 9.027print(extract_answer("so 1,250 + 30 = 1,280")) # 1280.0Use when
- +Large gains on multi-step maths, symbolic and multi-hop tasks for capable models
- +No training: exemplars or one instruction
- +Intermediate steps make many errors easy to spot and test
- +Composes with self-consistency, tool use (ReAct prompting) and verifiers
Avoid when
- −10–100× more output tokens: higher latency and cost
- −Little or negative gain for small models and single-step tasks
- −Chains can be fluent and wrong; one slip propagates to the answer
- −Needs answer extraction; free-form chains break naive parsers (pair with Structured output (JSON mode))
Check yourself
Under the error model, a model writes each step correctly with p = 0.9 but composes steps internally with a = 0.95. What does CoT do on a 6-step problem?
Check yourself
Why does zero-shot CoT (Kojima et al.) use a second prompt after the reasoning?
Remember
- ✓CoT makes the model write intermediate steps, so each step can read the previous results from context.
- ✓Few-shot CoT uses exemplars with worked reasoning. Zero-shot CoT uses "Let's think step by step" plus an answer-extraction prompt.
- ✓Gains grow with the number of dependent steps, roughly as , and appear only in sufficiently capable models.
- ✓The cost is output tokens and latency. Skip CoT for single-step tasks.
- ✓The chain is not a faithful explanation. Verify answers instead of trusting the reasoning text.