# Code generation

> Code LLMs write programs from context; because code can be run, they are scored by pass@k = 1 − C(n−c, k)/C(n, k) on unit tests, and improved by sampling many candidates, filtering with tests and repairing from errors.

Source: https://www.mlfoundry.in/learn/code-generation  
Section: Generative AI › Applications  
Level: core · 8 min read · updated 2026-10-21

**Code generation:** is the use of a language model to write source code from a natural-language request, surrounding code, or both. Codex (Chen et al., 2021, "Evaluating Large Language Models Trained on Code") set the template. It is a GPT model fine-tuned on public GitHub code, evaluated on the HumanEval benchmark of 164 hand-written Python problems with unit tests, and scored by functional correctness with the **pass@k** metric.

Code is the generative task where outputs can be checked mechanically. A program either passes its tests or it does not, so code generation is evaluated by execution rather than by text overlap. A sample that matches the reference word for word but has an off-by-one bug scores high on BLEU (see **BLEU, ROUGE and METEOR**) and fails every test. The same property drives the methods: sample several programs, run them, keep or repair what fails.

Three product shapes exist. **Inline completion** in the editor predicts the next lines from the code around the cursor, under a latency budget of a few hundred milliseconds. **Chat generation** writes a function, test or script from an instruction. **Agentic coding** edits a repository over many steps: it reads files, runs tests and proposes a patch, using the loop from **The agent loop** and **Tool use & function calling**. Generating queries from questions is covered in **Text-to-SQL**.

## Context: fill-in-the-middle

A left-to-right model sees only the code above the cursor, but the code below matters too: the return statement, later calls, the closing brace. Bavarian et al. (2022, "Efficient Training of Language Models to Fill in the Middle") cut training documents into prefix, middle and suffix, and moved the middle to the end with sentinel tokens:

$$
\text{doc}=(\text{pre},\,\text{mid},\,\text{suf})\;\longmapsto\;\langle\text{PRE}\rangle\,\text{pre}\,\langle\text{SUF}\rangle\,\text{suf}\,\langle\text{MID}\rangle\,\text{mid}\,\langle\text{EOT}\rangle
$$

- $\langle\text{PRE}\rangle,\langle\text{SUF}\rangle,\langle\text{MID}\rangle$: sentinel tokens; the model learns to generate the middle after seeing both sides
- $\text{PSM}$: this prefix–suffix–middle order; the paper also uses an SPM variant

The paper found that training with this transformation applied to a large fraction of documents leaves ordinary left-to-right ability essentially unchanged, which it called "FIM for free". StarCoder, Code Llama (7B and 13B) and most completion models since are trained this way. At inference the editor sends the code before and after the cursor, plus retrieved context: imports, signatures from other files, recently opened tabs. Choosing that context within a small token budget is a retrieval problem in its own right (see **RAG architecture**).

> Figure: The editor's code around the cursor becomes a fill-in-the-middle prompt plus retrieved repository context. Three samples of the missing middle of median() go to a sandbox with two unit tests. A averages the two middle elements and passes both. B returns 3 instead of 2.5 for an even-length list. C crashes with a TypeError because len(s)/2 is a float. Failures go back to the model with their error output. Below: with 10 samples of which 3 pass, pass@1 = 0.30 and pass@5 = 1 − 21/252 = 0.917.

## Measuring it: pass@k

pass@k is the probability that at least one of $k$ sampled programs passes all the tests. Drawing exactly $k$ samples per problem gives a high-variance estimate. Chen et al. instead draw $n\ge k$ samples (they used $n=200$), count the $c$ that pass, and compute the probability that a random subset of $k$ contains at least one of them:

$$
\text{pass@}k=\mathbb E_{\text{problems}}\!\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\right],\qquad \frac{\binom{n-c}{k}}{\binom{n}{k}}=\prod_{i=n-c+1}^{n}\left(1-\frac{k}{i}\right)
$$

- $n,\;c$: samples generated for the problem, and how many pass every test
- $k$: attempts allowed; pass@1 is the expected success of one sample
- $\prod$: the numerically stable form used in the paper; the binomials themselves overflow for $n=200$

The naive plug-in estimate $1-(1-c/n)^k$ is biased low. For $n=20$, $c=2$, $k=5$ it gives 0.410 against the unbiased 0.447. Report $n$, the temperature and the sampling method with any pass@k number, because all three change it.

Widely used benchmarks and well-known reported results (check the papers for exact settings)

| Benchmark / system | What it contains or reports |
| --- | --- |
| HumanEval (Chen et al., 2021) | 164 hand-written Python functions with docstrings, about 7.7 unit tests each |
| MBPP (Austin et al., 2021) | 974 crowd-sourced entry-level Python tasks, 3 test asserts each |
| APPS (Hendrycks et al., 2021) | 10,000 competitive-programming style problems in three difficulty levels |
| SWE-bench (Jimenez et al., 2023) | 2,294 real GitHub issues from 12 Python repositories; a patch must make the repo's tests pass. SWE-bench Verified (2024) is a human-checked subset of 500 |
| Codex 12B on HumanEval | 28.8% pass@1 and 70.2% pass@100; best temperature about 0.2 for pass@1 and 0.8 for pass@100 |
| AlphaCode (Li et al., 2022) | samples very many programs per problem, filters on the example tests, clusters and submits 10; average ranking in the top 54.3% in Codeforces contests |

## Sampling, temperature and filtering

Temperature (see **Temperature sampling**) trades the two ends of pass@k against each other. At low $T$ every sample is nearly the same program, so a problem is solved by all samples or by none. pass@1 is high, but extra samples add little. At higher $T$ the samples differ, so some hard problems get at least one correct program, at the price of more broken code per sample. A system that can verify candidates should sample at a higher temperature and select. A system that shows one suggestion should sample at a low one.

> Figure: 80 synthetic problems × 200 samples; a sample is correct with probability σ(z/T − 2T). Defaults (T = 0.20, k = 10, false pass 0.05): pass@1 33.0%, pass@10 48.0%, pass@100 62.9%; 10 problems are always solved, 27 never and 43 sometimes. The best temperature rises with k: 0.15 for pass@1, 0.65 for pass@10, 0.95 for pass@100. At T = 0.80: pass@1 17.4%, pass@100 90.5%. Filtering 100 samples by example tests gives 62.5% with one submission, 87.5% with perfect tests and 35.0% at false pass 0.30.

Filtering converts many samples into one answer without a human. This is the CodeT and AlphaCode idea: run candidates on example or model-generated tests and submit one that passes. It works only as well as the tests. When wrong programs often pass, the filter stops at the first lucky one, and more samples stop helping. Execution feedback also enables **self-repair**: put the failing test and traceback into the prompt and ask for a fix.

## Worked example: pass@k by hand

Four problems, n = 10 samples each; pass@k per problem is 1 − C(10−c, k)/C(10, k)

| Problem | c correct | pass@1 | pass@5 |
| --- | --- | --- | --- |
| P1 | 10 | 1 − 0 = 1.000 | 1.000 |
| P2 | 3 | 1 − 7/10 = 0.300 | 1 − 21/252 = 0.917 |
| P3 | 1 | 1 − 9/10 = 0.100 | 1 − 126/252 = 0.500 |
| P4 | 0 | 0.000 | 0.000 |
| Benchmark | mean | 1.400 / 4 = 0.350 | 2.417 / 4 = 0.604 |

```python
import subprocess, sys, tempfile
import numpy as np

def pass_at_k(n, c, k):
    """Unbiased pass@k (Chen et al., 2021), numerically stable form."""
    if n - c < k:
        return 1.0
    return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

def passes(program, tests, timeout=5):
    """Run a candidate plus its asserts in a fresh interpreter; True if exit code is 0."""
    with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False) as f:
        f.write(program + "\n\n" + tests)
    try:
        return subprocess.run([sys.executable, f.name], capture_output=True, timeout=timeout).returncode == 0
    except subprocess.TimeoutExpired:
        return False

head = "def median(xs):\n    s = sorted(xs)\n"
middles = ["    n = len(s)\n    m = (s[(n-1)//2] + s[n//2]) / 2\n",   # A
           "    m = s[len(s)//2]\n",                                   # B
           "    m = s[len(s)/2]\n"]                                    # C
tests = "assert median([3, 1, 2]) == 2\nassert median([4, 1, 3, 2]) == 2.5\n"
print([passes(head + mid + "    return m", tests) for mid in middles])  # [True, False, False]
print(round(pass_at_k(10, 3, 1), 3), round(pass_at_k(10, 3, 5), 3))     # 0.3 0.917
```

Unbiased pass@k and a minimal test runner (runs as is). A subprocess is not a security sandbox: run untrusted code in a container or VM with no network and resource limits.

> **Passing tests is not the same as safe or correct:** Generated code can pass its tests and still contain SQL injection, path traversal or hard-coded secrets. Pearce et al. (2021, "Asleep at the Keyboard?") found about 40% of Copilot completions in security-relevant scenarios were vulnerable. Models also invent package names that do not exist, which attackers can register. Pin dependencies, run static analysis and secret scanners, and review generated code like a junior colleague's pull request.

> **Contamination:** HumanEval and MBPP have been public since 2021, and their solutions are spread across the web that later models train on. High scores can partly reflect memorisation. Benchmarks such as LiveCodeBench collect problems published after a model's training cutoff to reduce this. Your own held-out tasks, from your codebase, remain the most informative test (see **LLM evaluation and benchmarking**).

- **Inline completion:** FIM models in the editor; latency and acceptance rate matter more than pass@1.
- **Test and boilerplate generation:** unit tests, docstrings, type stubs, migrations: high volume, easy to review.
- **Code translation and upgrades:** porting between languages or library versions, checked by the existing test suite.
- **Repository agents:** issue-to-patch loops scored on SWE-bench-style tasks, with tests as the stopping signal.

**Question:** A model produces n = 20 samples for a problem and c = 2 pass. What is the unbiased pass@5?

- A. 0.100
- B. 0.410
- C. 0.447
- D. 0.500

**Answer:** C. 0.447 — $1-\binom{18}{5}/\binom{20}{5}=1-8568/15504=0.447$. The plug-in estimate $1-(1-0.1)^5=0.410$ is biased low, and $c/n=0.1$ is pass@1, not pass@5.

**Question:** Your assistant shows the user one suggestion, while your batch pipeline generates 50 candidates and keeps one that passes the tests. How should the sampling temperatures differ?

- A. Both should use T = 0
- B. Low for the single suggestion; higher for the pipeline, which benefits from diverse candidates and can verify them
- C. High for the single suggestion, to be creative; low for the pipeline
- D. Temperature does not affect pass@k

**Answer:** B. Low for the single suggestion; higher for the pipeline, which benefits from diverse candidates and can verify them — pass@1 is maximised by near-greedy sampling. pass@k for large $k$ needs diversity, so its best temperature is higher: 0.2 against 0.8 for Codex, and 0.15 against 0.95 in the simulation. The pipeline can afford broken samples because tests filter them out.

**Key points**

- Code can be executed, so code generation is judged by functional correctness on unit tests, not by text similarity.
- pass@k $=\mathbb E[1-\binom{n-c}{k}/\binom{n}{k}]$ from $n\ge k$ samples; report $n$ and the temperature.
- Fill-in-the-middle training lets completion models use the code after the cursor at no cost to left-to-right ability.
- Low temperature maximises pass@1; higher temperature maximises pass@k for large k.
- Sample, execute, filter and repair: test-based selection only works as well as the tests.
- Generated code needs sandboxed execution, dependency checks and security review.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
