# Chain-of-thought prompting

> Chain-of-thought prompting (Wei et al., 2022) makes a model write intermediate steps before its answer; with 8 worked exemplars PaLM 540B rose from 17.9% to 56.9% on GSM8K, at the cost of many more output tokens.

Source: https://www.mlfoundry.in/learn/chain-of-thought-prompting  
Section: Generative AI › Prompt Engineering  
Level: core · 8 min read · updated 2026-10-21

**Chain-of-thought (CoT) prompting:** is a prompting technique in which the model is led to generate a series of intermediate reasoning steps in natural language before it states the final answer. Wei et al. introduced it in "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (NeurIPS 2022) using few-shot exemplars whose answers spell out the reasoning. Kojima et al. ("Large Language Models are Zero-Shot Reasoners", NeurIPS 2022) showed that the single trigger "Let's think step by step" gets much of the effect with no exemplars.

The problem is multi-step reasoning: arithmetic word problems, symbolic manipulation, questions that chain several facts. In **Few-shot prompting** the exemplars show only question and answer, so the model must produce the answer token straight away. Every intermediate quantity has to be computed inside a single forward pass per output token. A transformer has a fixed number of layers, so the amount of serial computation available before the first answer token is bounded.

CoT moves that computation into the output. Each written step becomes context for the next one, so the model reads its own intermediate results instead of holding them implicitly. In a product, CoT sits in the prompt (exemplars or an instruction), and an answer parser sits after generation. The parser pulls the final answer out of the reasoning text.

## Two ways to elicit it

> Figure: Top: the exemplar and outputs of Wei et al. (2022), Figure 1. The same model, given an exemplar that shows only "The answer is 11", answers the cafeteria question with 27. Given an exemplar that writes out 2 × 3 = 6 and 5 + 6 = 11, it writes 23 − 20 = 3, 3 + 6 = 9 and answers 9. Bottom: zero-shot CoT (Kojima et al., 2022) uses two calls, one to generate the reasoning after the trigger phrase and one to extract the answer in a fixed format.

**Variants used in practice**

1. **Few-shot CoT.** Put $k$ exemplars (Wei et al. used 8 for the maths benchmarks) of the form question → reasoning → "The answer is X." before the new question. The exemplars fix both the reasoning style and the answer format.
2. **Zero-shot CoT.** Append "Let's think step by step." to the answer prefix, generate, then send a second prompt such as "Therefore, the answer (arabic numerals) is" to extract a clean answer.
3. **Instruction CoT.** In chat models an instruction like "Reason step by step, then give the final answer on a line starting with Answer:" plays the same role and fixes the parse target.
4. **Structured decomposition.** Least-to-most prompting (Zhou et al., 2022) first asks for subproblems, then solves them in order. Program-aided variants (PAL, Gao et al., 2022; Program of Thoughts, Chen et al., 2022) have the model write Python and let an interpreter do the arithmetic.

## Why it helps, and when it does not

One way to read CoT is as a latent variable. The model's answer distribution marginalises over reasoning paths $r$:

$$
p(a\mid q)=\sum_{r} p(r\mid q)\,p(a\mid q,r)
$$

- $q$: the question plus the prompt (exemplars or trigger)
- $r$: a reasoning path, i.e. the generated chain of steps
- $a$: the final answer extracted after the chain

A single greedy CoT run samples one path. **Self-consistency** samples many and takes a majority vote over $a$, which approximates the sum. **Tree of thoughts** searches over partial paths instead of sampling whole ones.

A simple error model shows why CoT pays off on long problems. Suppose the model performs each written step correctly with probability $p$. Answering directly, it must compose all $n$ steps internally and succeeds with a lower per-step rate $a<p$:

$$
P_{\text{CoT}}(n)\approx p^{\,n},\qquad P_{\text{direct}}(n)\approx a^{\,n},\qquad \frac{P_{\text{CoT}}}{P_{\text{direct}}}=\left(\frac{p}{a}\right)^{n}
$$

- $n$: number of dependent reasoning steps
- $p$: probability that one written step is right
- $a$: effective per-step success when the steps stay implicit

The ratio grows exponentially with $n$ when $p>a$. If the model's written steps are worse than its intuition ($p<a$), CoT hurts. Wei et al. saw exactly this. Below roughly 100B parameters, models wrote fluent but illogical chains, and CoT gave no gain or lowered accuracy. This is one of the standard examples in **Emergent abilities**.

**Reported results (checkable in the papers)**

- **Few-shot CoT, GSM8K:** PaLM 540B: 17.9% with standard prompting, 56.9% with 8 CoT exemplars (Wei et al., 2022)
- **Zero-shot CoT:** text-davinci-002 with "Let's think step by step": MultiArith 17.7% → 78.7%, GSM8K 10.4% → 40.7% (Kojima et al., 2022)
- **Scale:** gains appeared only at about 10¹¹ parameters (LaMDA, GPT-3, PaLM families); smaller models were flat or worse
- **Faithfulness:** Turpin et al. (2023) biased few-shot prompts (e.g. the right option always "(A)"); models followed the bias while their written CoT never mentioned it

## Simulate it

> Figure: 200 seeded arithmetic problems per length n, solved by a toy model in two modes. CoT steps are right with probability p and a slip propagates; the direct answer is right with probability aⁿ. Default (large model, p = 0.96, a = 0.72, n = 4): CoT 169/200 (84.5%, pⁿ = 84.9%), direct 42/200 (21.0%, aⁿ = 26.9%). The CoT trace costs about 16.8 tokens against about 5 for a bare answer. Small model (p = 0.55, a = 0.65): at n = 4, CoT 19/200 against direct 32/200, so CoT hurts. Large model at n = 8: 143 against 11.

## Worked example: accuracy against cost

The error model with p = 0.96, a = 0.72, and latency = TTFT + output tokens × time per output token, with TTFT = 0.3 s and 25 ms per token.

| Quantity | Direct answer | Chain of thought |
| --- | --- | --- |
| Accuracy at n = 1 | 0.72¹ = 72.0% | 0.96¹ = 96.0% |
| Accuracy at n = 4 | 0.72⁴ = 26.9% | 0.96⁴ = 84.9% |
| Accuracy at n = 8 | 0.72⁸ = 7.2% | 0.96⁸ = 72.1% |
| Output tokens | 5 | 100 (typical short chain) |
| Latency | 0.3 + 5 × 0.025 = 0.425 s | 0.3 + 100 × 0.025 = 2.8 s |

At $n=4$ CoT turns a 26.9% task into an 84.9% one, for about 6.6× the latency and 20× the output tokens. At $n=1$ the gain is 24 points for the same price. For single-step lookups or classification, that price is often not worth paying. See **Cost and latency optimization**.

```python
import re

EXEMPLAR = ("Q: Roger has 5 tennis balls. He buys 2 more cans of tennis balls. Each can has "
            "3 tennis balls. How many tennis balls does he have now?\n"
            "A: Roger started with 5 balls. 2 cans of 3 tennis balls each is 6 tennis balls. "
            "5 + 6 = 11. The answer is 11.\n\n")

def few_shot_cot(question):
    return EXEMPLAR + f"Q: {question}\nA:"

def zero_shot_cot(question, complete):
    """Kojima et al. (2022): reasoning extraction, then answer extraction."""
    stage1 = f"Q: {question}\nA: Let's think step by step."
    reasoning = complete(stage1)
    stage2 = stage1 + reasoning + "\nTherefore, the answer (arabic numerals) is"
    return reasoning, complete(stage2)

def extract_answer(text):
    """Prefer 'The answer is N'; fall back to the last number in the text."""
    m = re.search(r"answer is\s*\$?(-?[\d,]*\.?\d+)", text)
    nums = [m.group(1)] if m else re.findall(r"-?[\d,]*\.?\d+", text)
    return float(nums[-1].replace(",", "")) if nums else None

out = ("The cafeteria had 23 apples originally. They used 20 to make lunch. So they had "
       "23 - 20 = 3. They bought 6 more apples, so they have 3 + 6 = 9. The answer is 9.")
print(extract_answer(out))                       # 9.0
print(extract_answer("so 1,250 + 30 = 1,280"))   # 1280.0
```

Few-shot and zero-shot CoT prompts with an answer extractor. `complete` stands for any text-completion call; the extractor was run on the sample outputs shown (prints 9.0 and 1280.0).

> **The chain is not an explanation:** A CoT is text the model generated, not a trace of its computation. Turpin et al. (2023) showed that models can reach an answer for a reason the chain never mentions. Do not use the chain as an audit log or as proof of correctness. Check the final answer (unit tests, a calculator, a verifier). Do not show raw chains to users as justification.

> **Reasoning models:** Since late 2024, reasoning models (for example OpenAI o1, released September 2024, and DeepSeek-R1, January 2025) are trained with reinforcement learning to produce long chains on their own (see **GRPO**). For them, a "think step by step" prompt adds little. Some APIs hide or summarise the chain and still bill its tokens.

**Strengths**

- Large gains on multi-step maths, symbolic and multi-hop tasks for capable models
- No training: exemplars or one instruction
- Intermediate steps make many errors easy to spot and test
- Composes with self-consistency, tool use (**ReAct prompting**) and verifiers

**Limitations**

- 10–100× more output tokens: higher latency and cost
- Little or negative gain for small models and single-step tasks
- Chains can be fluent and wrong; one slip propagates to the answer
- Needs answer extraction; free-form chains break naive parsers (pair with **Structured output (JSON mode)**)

**Question:** Under the error model, a model writes each step correctly with p = 0.9 but composes steps internally with a = 0.95. What does CoT do on a 6-step problem?

- A. Helps: 0.9⁶ > 0.95⁶
- B. Hurts: 0.9⁶ ≈ 53% against 0.95⁶ ≈ 74%
- C. No effect, because n is the same in both modes
- D. Helps, because longer outputs always mean more computation

**Answer:** B. Hurts: 0.9⁶ ≈ 53% against 0.95⁶ ≈ 74% — With $p<a$ the ratio $(p/a)^n=(0.9/0.95)^6\approx0.72$ is below 1. Writing steps only helps if the written steps are more reliable than the implicit ones. That is what Wei et al. saw for small models.

**Question:** Why does zero-shot CoT (Kojima et al.) use a second prompt after the reasoning?

- A. To make the reasoning longer
- B. To pull a clean answer in a fixed format out of free-form reasoning
- C. To check the reasoning for hallucinations
- D. Because the first call cannot see the question

**Answer:** B. To pull a clean answer in a fixed format out of free-form reasoning — Without exemplars nothing fixes the answer format. The extraction prompt ("Therefore, the answer (arabic numerals) is") turns the reasoning into a parseable answer.

**Key points**

- CoT makes the model write intermediate steps, so each step can read the previous results from context.
- Few-shot CoT uses exemplars with worked reasoning. Zero-shot CoT uses "Let's think step by step" plus an answer-extraction prompt.
- Gains grow with the number of dependent steps, roughly as $(p/a)^n$, and appear only in sufficiently capable models.
- The cost is output tokens and latency. Skip CoT for single-step tasks.
- The chain is not a faithful explanation. Verify answers instead of trusting the reasoning text.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
