Definition
Code generation is the use of a language model to write source code from a natural-language request, surrounding code, or both. Codex (Chen et al., 2021, "Evaluating Large Language Models Trained on Code") set the template. It is a GPT model fine-tuned on public GitHub code, evaluated on the HumanEval benchmark of 164 hand-written Python problems with unit tests, and scored by functional correctness with the pass@k metric.
Code is the generative task where outputs can be checked mechanically. A program either passes its tests or it does not, so code generation is evaluated by execution rather than by text overlap. A sample that matches the reference word for word but has an off-by-one bug scores high on BLEU (see BLEU, ROUGE and METEOR) and fails every test. The same property drives the methods: sample several programs, run them, keep or repair what fails.
Three product shapes exist. Inline completion in the editor predicts the next lines from the code around the cursor, under a latency budget of a few hundred milliseconds. Chat generation writes a function, test or script from an instruction. Agentic coding edits a repository over many steps: it reads files, runs tests and proposes a patch, using the loop from The agent loop and Tool use & function calling. Generating queries from questions is covered in Text-to-SQL.
Context: fill-in-the-middle
A left-to-right model sees only the code above the cursor, but the code below matters too: the return statement, later calls, the closing brace. Bavarian et al. (2022, "Efficient Training of Language Models to Fill in the Middle") cut training documents into prefix, middle and suffix, and moved the middle to the end with sentinel tokens:
- sentinel tokens; the model learns to generate the middle after seeing both sides
- this prefix–suffix–middle order; the paper also uses an SPM variant
The paper found that training with this transformation applied to a large fraction of documents leaves ordinary left-to-right ability essentially unchanged, which it called "FIM for free". StarCoder, Code Llama (7B and 13B) and most completion models since are trained this way. At inference the editor sends the code before and after the cursor, plus retrieved context: imports, signatures from other files, recently opened tabs. Choosing that context within a small token budget is a retrieval problem in its own right (see RAG architecture).
Measuring it: pass@k
pass@k is the probability that at least one of sampled programs passes all the tests. Drawing exactly samples per problem gives a high-variance estimate. Chen et al. instead draw samples (they used ), count the that pass, and compute the probability that a random subset of contains at least one of them:
- samples generated for the problem, and how many pass every test
- attempts allowed; pass@1 is the expected success of one sample
- the numerically stable form used in the paper; the binomials themselves overflow for
The naive plug-in estimate is biased low. For , , it gives 0.410 against the unbiased 0.447. Report , the temperature and the sampling method with any pass@k number, because all three change it.
| Benchmark / system | What it contains or reports |
|---|---|
| HumanEval (Chen et al., 2021) | 164 hand-written Python functions with docstrings, about 7.7 unit tests each |
| MBPP (Austin et al., 2021) | 974 crowd-sourced entry-level Python tasks, 3 test asserts each |
| APPS (Hendrycks et al., 2021) | 10,000 competitive-programming style problems in three difficulty levels |
| SWE-bench (Jimenez et al., 2023) | 2,294 real GitHub issues from 12 Python repositories; a patch must make the repo's tests pass. SWE-bench Verified (2024) is a human-checked subset of 500 |
| Codex 12B on HumanEval | 28.8% pass@1 and 70.2% pass@100; best temperature about 0.2 for pass@1 and 0.8 for pass@100 |
| AlphaCode (Li et al., 2022) | samples very many programs per problem, filters on the example tests, clusters and submits 10; average ranking in the top 54.3% in Codeforces contests |
Sampling, temperature and filtering
Temperature (see Temperature sampling) trades the two ends of pass@k against each other. At low every sample is nearly the same program, so a problem is solved by all samples or by none. pass@1 is high, but extra samples add little. At higher the samples differ, so some hard problems get at least one correct program, at the price of more broken code per sample. A system that can verify candidates should sample at a higher temperature and select. A system that shows one suggestion should sample at a low one.
Filtering converts many samples into one answer without a human. This is the CodeT and AlphaCode idea: run candidates on example or model-generated tests and submit one that passes. It works only as well as the tests. When wrong programs often pass, the filter stops at the first lucky one, and more samples stop helping. Execution feedback also enables self-repair: put the failing test and traceback into the prompt and ask for a fix.
Worked example: pass@k by hand
| Problem | c correct | pass@1 | pass@5 |
|---|---|---|---|
| P1 | 10 | 1 − 0 = 1.000 | 1.000 |
| P2 | 3 | 1 − 7/10 = 0.300 | 1 − 21/252 = 0.917 |
| P3 | 1 | 1 − 9/10 = 0.100 | 1 − 126/252 = 0.500 |
| P4 | 0 | 0.000 | 0.000 |
| Benchmark | mean | 1.400 / 4 = 0.350 | 2.417 / 4 = 0.604 |
1import subprocess, sys, tempfile2import numpy as np34def pass_at_k(n, c, k):5 """Unbiased pass@k (Chen et al., 2021), numerically stable form."""6 if n - c < k:7 return 1.08 return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))910def passes(program, tests, timeout=5):11 """Run a candidate plus its asserts in a fresh interpreter; True if exit code is 0."""12 with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:13 f.write(program + "\n\n" + tests)14 try:15 return subprocess.run([sys.executable, f.name], capture_output=True, timeout=timeout).returncode == 016 except subprocess.TimeoutExpired:17 return False1819head = "def median(xs):\n s = sorted(xs)\n"20middles = [" n = len(s)\n m = (s[(n-1)//2] + s[n//2]) / 2\n", # A21 " m = s[len(s)//2]\n", # B22 " m = s[len(s)/2]\n"] # C23tests = "assert median([3, 1, 2]) == 2\nassert median([4, 1, 3, 2]) == 2.5\n"24print([passes(head + mid + " return m", tests) for mid in middles]) # [True, False, False]25print(round(pass_at_k(10, 3, 1), 3), round(pass_at_k(10, 3, 5), 3)) # 0.3 0.917Where it's used
Inline completion
FIM models in the editor; latency and acceptance rate matter more than pass@1.
Test and boilerplate generation
unit tests, docstrings, type stubs, migrations: high volume, easy to review.
Code translation and upgrades
porting between languages or library versions, checked by the existing test suite.
Repository agents
issue-to-patch loops scored on SWE-bench-style tasks, with tests as the stopping signal.
Check yourself
A model produces n = 20 samples for a problem and c = 2 pass. What is the unbiased pass@5?
Check yourself
Your assistant shows the user one suggestion, while your batch pipeline generates 50 candidates and keeps one that passes the tests. How should the sampling temperatures differ?
Remember
- ✓Code can be executed, so code generation is judged by functional correctness on unit tests, not by text similarity.
- ✓pass@k from samples; report and the temperature.
- ✓Fill-in-the-middle training lets completion models use the code after the cursor at no cost to left-to-right ability.
- ✓Low temperature maximises pass@1; higher temperature maximises pass@k for large k.
- ✓Sample, execute, filter and repair: test-based selection only works as well as the tests.
- ✓Generated code needs sandboxed execution, dependency checks and security review.