Code generation

Code LLMs write programs from context; because code can be run, they are scored by pass@k = 1 − C(n−c, k)/C(n, k) on unit tests, and improved by sampling many candidates, filtering with tests and repairing from errors.

8 min read836 wordscore0 viewsupdated 2026-10-21

Definition

Code generation is the use of a language model to write source code from a natural-language request, surrounding code, or both. Codex (Chen et al., 2021, "Evaluating Large Language Models Trained on Code") set the template. It is a GPT model fine-tuned on public GitHub code, evaluated on the HumanEval benchmark of 164 hand-written Python problems with unit tests, and scored by functional correctness with the pass@k metric.

Code is the generative task where outputs can be checked mechanically. A program either passes its tests or it does not, so code generation is evaluated by execution rather than by text overlap. A sample that matches the reference word for word but has an off-by-one bug scores high on BLEU (see BLEU, ROUGE and METEOR) and fails every test. The same property drives the methods: sample several programs, run them, keep or repair what fails.

Three product shapes exist. Inline completion in the editor predicts the next lines from the code around the cursor, under a latency budget of a few hundred milliseconds. Chat generation writes a function, test or script from an instruction. Agentic coding edits a repository over many steps: it reads files, runs tests and proposes a patch, using the loop from The agent loop and Tool use & function calling. Generating queries from questions is covered in Text-to-SQL.

Context: fill-in-the-middle

A left-to-right model sees only the code above the cursor, but the code below matters too: the return statement, later calls, the closing brace. Bavarian et al. (2022, "Efficient Training of Language Models to Fill in the Middle") cut training documents into prefix, middle and suffix, and moved the middle to the end with sentinel tokens:

doc=(pre, mid, suf)  ⟼  ⟨PRE⟩ pre ⟨SUF⟩ suf ⟨MID⟩ mid ⟨EOT⟩\text{doc}=(\text{pre},\,\text{mid},\,\text{suf})\;\longmapsto\;\langle\text{PRE}\rangle\,\text{pre}\,\langle\text{SUF}\rangle\,\text{suf}\,\langle\text{MID}\rangle\,\text{mid}\,\langle\text{EOT}\rangle
⟨PRE⟩,⟨SUF⟩,⟨MID⟩\langle\text{PRE}\rangle,\langle\text{SUF}\rangle,\langle\text{MID}\rangle
sentinel tokens; the model learns to generate the middle after seeing both sides
PSM\text{PSM}
this prefix–suffix–middle order; the paper also uses an SPM variant

The paper found that training with this transformation applied to a large fraction of documents leaves ordinary left-to-right ability essentially unchanged, which it called "FIM for free". StarCoder, Code Llama (7B and 13B) and most completion models since are trained this way. At inference the editor sends the code before and after the cursor, plus retrieved context: imports, signatures from other files, recently opened tabs. Choosing that context within a small token budget is a retrieval problem in its own right (see RAG architecture).

context → sample → execute → repaireditor, cursor at ▌def median(xs): """Median of a list.""" s = sorted(xs) ▌ return mfill-in-the-middle prompt+ retrieved repo context (imports, signatures from utils.py, open tabs)<PRE> def median ... sorted(xs)<SUF> return m<MID> → model writes the middlesample k = 3, T = 0.8sample A ✓ passes 2/2n = len(s)m = (s[(n-1)//2] + s[n//2]) / 2sample B ✗ fails test 2m = s[len(s)//2]median([4,1,3,2]) → 3, not 2.5sample C ✗ crashesm = s[len(s)/2]TypeError: list indices ...sandbox: run hidden or generated unit tests (time and memory limits, no network)assert median([3, 1, 2]) == 2assert median([4, 1, 3, 2]) == 2.5pass → show A to the user(or open a pull request)fail → append test output / tracebackto the prompt and ask for a fix (self-repair)scoring a benchmark: n = 10 samples for one problem, c = 3 pass all testspass@1 = 1 − C(7,1)/C(10,1) = 1 − 7/10 = 0.30pass@5 = 1 − C(7,5)/C(10,5) = 1 − 21/252 = 0.917probability that at least one ofk drawn samples is correct
fig.The editor's code around the cursor becomes a fill-in-the-middle prompt plus retrieved repository context. Three samples of the missing middle of median() go to a sandbox with two unit tests. A averages the two middle elements and passes both. B returns 3 instead of 2.5 for an even-length list. C crashes with a TypeError because len(s)/2 is a float. Failures go back to the model with their error output. Below: with 10 samples of which 3 pass, pass@1 = 0.30 and pass@5 = 1 − 21/252 = 0.917.

Measuring it: pass@k

pass@k is the probability that at least one of kk sampled programs passes all the tests. Drawing exactly kk samples per problem gives a high-variance estimate. Chen et al. instead draw n≥kn\ge k samples (they used n=200n=200), count the cc that pass, and compute the probability that a random subset of kk contains at least one of them:

pass@k=Eproblems ⁣[1−(n−ck)(nk)],(n−ck)(nk)=∏i=n−c+1n(1−ki)\text{pass@}k=\mathbb E_{\text{problems}}\!\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\right],\qquad \frac{\binom{n-c}{k}}{\binom{n}{k}}=\prod_{i=n-c+1}^{n}\left(1-\frac{k}{i}\right)
n,  cn,\;c
samples generated for the problem, and how many pass every test
kk
attempts allowed; pass@1 is the expected success of one sample
∏\prod
the numerically stable form used in the paper; the binomials themselves overflow for n=200n=200

The naive plug-in estimate 1−(1−c/n)k1-(1-c/n)^k is biased low. For n=20n=20, c=2c=2, k=5k=5 it gives 0.410 against the unbiased 0.447. Report nn, the temperature and the sampling method with any pass@k number, because all three change it.

Benchmark / systemWhat it contains or reports
HumanEval (Chen et al., 2021)164 hand-written Python functions with docstrings, about 7.7 unit tests each
MBPP (Austin et al., 2021)974 crowd-sourced entry-level Python tasks, 3 test asserts each
APPS (Hendrycks et al., 2021)10,000 competitive-programming style problems in three difficulty levels
SWE-bench (Jimenez et al., 2023)2,294 real GitHub issues from 12 Python repositories; a patch must make the repo's tests pass. SWE-bench Verified (2024) is a human-checked subset of 500
Codex 12B on HumanEval28.8% pass@1 and 70.2% pass@100; best temperature about 0.2 for pass@1 and 0.8 for pass@100
AlphaCode (Li et al., 2022)samples very many programs per problem, filters on the example tests, clusters and submits 10; average ranking in the top 54.3% in Codeforces contests
Widely used benchmarks and well-known reported results (check the papers for exact settings)

Sampling, temperature and filtering

Temperature (see Temperature sampling) trades the two ends of pass@k against each other. At low TT every sample is nearly the same program, so a problem is solved by all samples or by none. pass@1 is high, but extra samples add little. At higher TT the samples differ, so some hard problems get at least one correct program, at the price of more broken code per sample. A system that can verify candidates should sample at a higher temperature and select. A system that shows one suggestion should sample at a low one.

80 problems × 200 samples · T = 0.20 · pass@10 = 48.0% · filtered@1 from 10 = 47.5%success vs samples k (log scale)0255075100110100pass@k (any of k correct)filtered@1: submit first that passes testspass@1 / @10 / @100 vs temperature0.10.511.5best T: @1 0.15 · @10 0.65 · @100 0.95fraction of the 200 samples that are correct, per problem (sorted) at T = 0.20all correct: 10 · none correct: 27 · mixed: 43pass@1 33.0% · pass@10 48.0% · pass@100 62.9%pass@k = mean over problems of 1 − C(n−c, k)/C(n, k), n = 200 samples, c correctcorrect w.p. σ(z/T − 2T): low T → all-or-nothing per problem; high T → diverse, more broken codea wrong program passes the example tests with probability 0.05; correct programs always pass
fig.80 synthetic problems × 200 samples; a sample is correct with probability σ(z/T − 2T). Defaults (T = 0.20, k = 10, false pass 0.05): pass@1 33.0%, pass@10 48.0%, pass@100 62.9%; 10 problems are always solved, 27 never and 43 sometimes. The best temperature rises with k: 0.15 for pass@1, 0.65 for pass@10, 0.95 for pass@100. At T = 0.80: pass@1 17.4%, pass@100 90.5%. Filtering 100 samples by example tests gives 62.5% with one submission, 87.5% with perfect tests and 35.0% at false pass 0.30.

Filtering converts many samples into one answer without a human. This is the CodeT and AlphaCode idea: run candidates on example or model-generated tests and submit one that passes. It works only as well as the tests. When wrong programs often pass, the filter stops at the first lucky one, and more samples stop helping. Execution feedback also enables self-repair: put the failing test and traceback into the prompt and ask for a fix.

Worked example: pass@k by hand

Problemc correctpass@1pass@5
P1101 − 0 = 1.0001.000
P231 − 7/10 = 0.3001 − 21/252 = 0.917
P311 − 9/10 = 0.1001 − 126/252 = 0.500
P400.0000.000
Benchmarkmean1.400 / 4 = 0.3502.417 / 4 = 0.604
Four problems, n = 10 samples each; pass@k per problem is 1 − C(10−c, k)/C(10, k)
python
1import subprocess, sys, tempfile
2import numpy as np
3
4def pass_at_k(n, c, k):
5 """Unbiased pass@k (Chen et al., 2021), numerically stable form."""
6 if n - c < k:
7 return 1.0
8 return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))
9
10def passes(program, tests, timeout=5):
11 """Run a candidate plus its asserts in a fresh interpreter; True if exit code is 0."""
12 with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
13 f.write(program + "\n\n" + tests)
14 try:
15 return subprocess.run([sys.executable, f.name], capture_output=True, timeout=timeout).returncode == 0
16 except subprocess.TimeoutExpired:
17 return False
18
19head = "def median(xs):\n s = sorted(xs)\n"
20middles = [" n = len(s)\n m = (s[(n-1)//2] + s[n//2]) / 2\n", # A
21 " m = s[len(s)//2]\n", # B
22 " m = s[len(s)/2]\n"] # C
23tests = "assert median([3, 1, 2]) == 2\nassert median([4, 1, 3, 2]) == 2.5\n"
24print([passes(head + mid + " return m", tests) for mid in middles]) # [True, False, False]
25print(round(pass_at_k(10, 3, 1), 3), round(pass_at_k(10, 3, 5), 3)) # 0.3 0.917
Unbiased pass@k and a minimal test runner (runs as is). A subprocess is not a security sandbox: run untrusted code in a container or VM with no network and resource limits.

Where it's used

Inline completion

FIM models in the editor; latency and acceptance rate matter more than pass@1.

Test and boilerplate generation

unit tests, docstrings, type stubs, migrations: high volume, easy to review.

Code translation and upgrades

porting between languages or library versions, checked by the existing test suite.

Repository agents

issue-to-patch loops scored on SWE-bench-style tasks, with tests as the stopping signal.

Check yourself

A model produces n = 20 samples for a problem and c = 2 pass. What is the unbiased pass@5?

Check yourself

Your assistant shows the user one suggestion, while your batch pipeline generates 50 candidates and keeps one that passes the tests. How should the sampling temperatures differ?

Remember

  • ✓Code can be executed, so code generation is judged by functional correctness on unit tests, not by text similarity.
  • ✓pass@k =E[1−(n−ck)/(nk)]=\mathbb E[1-\binom{n-c}{k}/\binom{n}{k}] from n≥kn\ge k samples; report nn and the temperature.
  • ✓Fill-in-the-middle training lets completion models use the code after the cursor at no cost to left-to-right ability.
  • ✓Low temperature maximises pass@1; higher temperature maximises pass@k for large k.
  • ✓Sample, execute, filter and repair: test-based selection only works as well as the tests.
  • ✓Generated code needs sandboxed execution, dependency checks and security review.
← back to Generative AI roadmap