# Privacy and data leakage

> Generative AI leaks private data two ways: memorised training text recovered by extraction or membership attacks (Carlini et al. 2021), and prompts, RAG indexes and logs at inference; DP-SGD bounds the first by (ε, δ), access control the second.

Source: https://www.mlfoundry.in/learn/privacy-and-data-leakage  
Section: Generative AI › Safety, Alignment and Responsible AI  
Level: core · 8 min read · updated 2026-10-21

**Privacy and data leakage:** in generative AI means private information reaching someone who should not see it, through the model or the system around it. Two families matter. **Training-data leakage**: the model memorises parts of its training set and reproduces them, shown at scale by Carlini et al. ("Extracting Training Data from Large Language Models", USENIX Security 2021). **Inference-time leakage**: prompts, system prompts, retrieved documents, embeddings and logs flow to the wrong user or a third party. OWASP's Top 10 for LLM Applications (2025) lists these as LLM02 Sensitive Information Disclosure, LLM07 System Prompt Leakage and LLM08 Vector and Embedding Weaknesses.

The two families need different fixes. Memorisation is a property of training: it is set by the data, its duplication and the optimiser, and only changes to training remove it. Inference-time leakage needs no memorisation at all. It comes from how the application moves data, so it is fixed with ordinary engineering: redaction, access control, retention limits and contracts.

## Memorisation and extraction

Carlini et al. (2021) sampled 600,000 generations from GPT-2, ranked them by how much more likely GPT-2 found them than a reference model or zlib compression did, and checked the top 1,800 against the training set. 604 were verbatim training text, including names, phone numbers, email addresses and code. Some of these strings appeared in only one training document.

Follow-up work measured what drives it. Carlini et al. ("Quantifying Memorization Across Neural Language Models", ICLR 2023) found extractable memorisation grows log-linearly with model size, with how often a string is duplicated, and with the length of the prompt prefix. Kandpal, Wallace and Raffel (ICML 2022) showed deduplicating the training set sharply reduces regenerated training text. Nasr et al. (2023) extracted training data from the production ChatGPT (gpt-3.5-turbo) with a "divergence" prompt asking it to repeat one word forever, after which it drifted into memorised text.

> Figure: Lane 1: a crawled email address, a planted canary ("my PIN is 7302") and a page duplicated 40 times enter training. Rare and repeated strings get memorised, so the released model completes "my PIN is" with "7302", and its low loss on a record reveals membership. Lane 2: no memorisation needed. A user pastes a date of birth and account number, the app holds a system prompt, a two-tenant RAG index, stored embeddings and logs, and everything passes through the model provider. Green ticks are the control at each stage.

Two attacks are measured. **Extraction** recovers a training string from the model. **Membership inference** (Shokri et al., IEEE S&P 2017) decides whether a given record was in the training set, usually by thresholding the model's loss on it. Report membership attacks as the true-positive rate at a low false-positive rate such as 0.1% or 1% (Carlini et al., "Membership Inference Attacks From First Principles", 2022). Average accuracy hides the few records that are exposed with high confidence.

$$
\text{exposure}(s)=\log_2|\mathcal R|-\log_2\operatorname{rank}_\theta(s)
$$

- $s$: a random **canary** inserted into the training data, e.g. "my PIN is 7302"
- $\mathcal R$: the space of possible canary values (here $10^4$ PINs)
- $\operatorname{rank}_\theta(s)$: position of the true canary when all candidates are sorted by model perplexity (1 = most likely)

Exposure comes from "The Secret Sharer" (Carlini et al., USENIX Security 2019). An exposure close to $\log_2|\mathcal R|$ means the model ranks the secret first and an attacker can read it off. Planting canaries and measuring their exposure is a cheap way to audit your own fine-tuning run.

Worked example: a 4-digit PIN canary, |R| = 10⁴, log₂ 10⁴ = 13.29 bits

| Rank of true PIN | log₂ rank | Exposure (bits) | Reading |
| --- | --- | --- | --- |
| 1 | 0 | 13.29 | maximum: the model's top guess is the secret |
| 3 | 1.58 | 11.70 | found within 3 guesses |
| 100 | 6.64 | 6.64 | partly memorised |
| 5,000 | 12.29 | 1.00 | no better than about half the candidates |

## Differential privacy

$$
\Pr[M(D)\in S]\;\le\;e^{\varepsilon}\,\Pr[M(D')\in S]+\delta\quad\text{for all } S \text{ and all } D,D' \text{ differing in one record}
$$

- $M$: the randomised training algorithm; its output is the model
- $\varepsilon$: privacy budget: smaller means one record changes the output distribution less
- $\delta$: small failure probability, chosen well below $1/N$

The definition is due to Dwork, McSherry, Nissim and Smith (2006). It bounds every attacker, including membership inference: whatever test is run, its true-positive and false-positive rates satisfy $\text{TPR}\le e^\varepsilon\,\text{FPR}+\delta$. DP-SGD (Abadi et al., CCS 2016) makes SGD private by clipping each example's gradient and adding Gaussian noise:

$$
\tilde g=\frac1B\Big(\sum_{i\in\text{batch}}\operatorname{clip}_C(g_i)+\mathcal N(0,\sigma^2C^2I)\Big),\qquad \mu\approx q\sqrt{T\left(e^{1/\sigma^2}-1\right)},\quad \text{TPR}\le\Phi\!\left(\Phi^{-1}(\text{FPR})+\mu\right)
$$

- $\operatorname{clip}_C$: rescale $g_i$ to norm at most $C$, so one example's influence is bounded
- $\sigma$: noise multiplier
- $q,\;T$: sampling rate $B/N$ and number of steps
- $\mu$: Gaussian-DP parameter (central-limit accountant, Bu, Dong, Long and Su 2020); it converts to an $(\varepsilon,\delta)$ pair

> Figure: DP-SGD on N = 60,000 examples with C = 1 and δ = 10⁻⁵. Defaults (σ = 1.1, B = 256, 60 epochs): T = 14,062 steps, μ = 0.574, ε = 2.32; the best possible membership attack reaches only 3.98% TPR at 1% FPR and 23.9% at 10%. Drop σ to 0.6: ε = 9.78 and TPR 35.9% at 1% FPR. Raise σ to 2.0: ε = 1.01. Keep σ = 1.1 but train 200 epochs: ε = 4.62. B = 1024 for the same 60 epochs: ε = 5.14.

Since $T=EN/B$, the formula reduces to $\mu=\sqrt{qE\,(e^{1/\sigma^2}-1)}$ for $E$ epochs. Privacy cost grows with the square root of the epochs and of the batch fraction. B = 64 for 60 epochs and B = 256 for 15 epochs both give μ = 0.287 and ε = 1.08. Larger batches still help utility, because the noise on the averaged gradient is $\sigma C/B$, so σ is usually raised with B.

1. Sampling rate: $q=256/60{,}000=0.004267$; steps $T=\lfloor 60\cdot60{,}000/256\rfloor=14{,}062$.
2. Noise term: $1/\sigma^2=1/1.21=0.8264$, $e^{0.8264}-1=1.2852$.
3. $\sqrt{T\cdot1.2852}=\sqrt{18{,}072}=134.43$; $\mu=0.004267\times134.43=0.574$.
4. Solve $\delta(\varepsilon)=\Phi(-\varepsilon/\mu+\mu/2)-e^{\varepsilon}\Phi(-\varepsilon/\mu-\mu/2)=10^{-5}$: $\varepsilon=2.32$.
5. Attack bound at FPR 1%: $\Phi(\Phi^{-1}(0.01)+0.574)=\Phi(-2.326+0.574)=\Phi(-1.752)=3.98\%$, against 1% for a random guess.

**Strengths**

- The only defence with a proven bound that holds against every attack, including future ones.
- Composes: the accountant tracks the budget across epochs and multiple releases.
- Fine-tuning a public pre-trained model with DP-SGD on private data loses far less accuracy than training from scratch with DP.

**Limitations**

- Per-example gradients cost memory and time; clipping and noise lower accuracy, most for rare classes and groups.
- Protects records, not facts: if a fact appears in many records it can still be learned and repeated.
- ε values used in practice (often 1–10) give weak formal guarantees at the top of that range.

## Inference-time leakage

Leak paths that need no memorisation, and the usual control

| Path | Example | Control |
| --- | --- | --- |
| Prompts to a third party | Users paste contracts or patient notes into a hosted model | Redact or pseudonymise first; contract for zero or short retention and no training on inputs |
| System-prompt leakage | Prompt injection makes the bot print its instructions, including an API key | Never put secrets in prompts; assume the system prompt is public (see **Prompt injection**) |
| Cross-tenant retrieval | One shared vector index returns tenant B's chunk to tenant A | Filter by tenant and user permissions at query time, not after generation |
| Embedding inversion | vec2text (Morris et al. 2023) reconstructs input text from its embedding | Treat stored embeddings as sensitive as the text they encode |
| Logs and traces | Full prompts kept indefinitely in observability tools | Minimise, mask, set retention; restrict access |

**Checkable specifics**

- **GPT-2 extraction (2021):** 600,000 samples → 1,800 candidates → 604 verbatim training sequences
- **Canary exposure:** maximum $\log_2|\mathcal R|$; 13.29 bits for a 4-digit PIN, 29.9 bits for a 9-digit number
- **DP accounting:** δ is set well below 1/N; DP-SGD papers commonly report ε between about 1 and 10
- **GDPR (EU 2016/679):** data minimisation (Art. 5(1)(c)), right to erasure (Art. 17), data protection by design (Art. 25) apply to personal data in prompts, logs and training sets

```python
import re

PATTERNS = [                                   # order matters: most specific first
    ("EMAIL", re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+")),
    ("DATE",  re.compile(r"\b\d{4}-\d{2}-\d{2}\b")),
    ("ACCT",  re.compile(r"\b\d{2}-\d{4}-\d{2}\b")),
    ("PHONE", re.compile(r"\+?\d[\d ()-]{7,}\d")),
]

def pseudonymise(text):
    """Replace PII with stable placeholders; return text and the reverse map."""
    mapping = {}
    for label, pat in PATTERNS:
        def sub(m):
            key = next((k for k, v in mapping.items() if v == m.group()), None)
            if key is None:
                key = f"<{label}_{sum(k.startswith('<' + label) for k in mapping) + 1}>"
                mapping[key] = m.group()
            return key
        text = pat.sub(sub, text)
    return text, mapping

def restore(text, mapping):
    for key, val in mapping.items():
        text = text.replace(key, val)
    return text

prompt = ("Draft a reply to J. Smith (j.smith@mail.example, +44 20 7946 0958), "
          "DOB 1984-03-02, about account 12-3456-78.")
safe, m = pseudonymise(prompt)
print(safe)
# Draft a reply to J. Smith (<EMAIL_1>, <PHONE_1>), DOB <DATE_1>, about account <ACCT_1>.
reply = "Dear J. Smith, we have updated account <ACCT_1>."   # what the API returns
print(restore(reply, m))
# Dear J. Smith, we have updated account 12-3456-78.
```

Pseudonymise PII before a prompt leaves your boundary and restore it in the reply (standard library only, runs as is). Regexes miss names such as "J. Smith"; production systems add an NER-based detector

> **Deleting the row does not delete the memory:** Removing a person's record from the dataset after training leaves whatever the weights memorised. Honouring an erasure request for a trained model means retraining without the record, or an approximate **machine unlearning** method whose effect you then test, for example with canary exposure or a membership attack. Scrub and deduplicate before training, when it is cheap.

> **Not legal advice:** Laws such as the GDPR are summarised here in general terms. Whether a model or a log counts as personal data, and what a given contract with a model provider permits, depends on facts and jurisdiction. Ask counsel.

**Question:** A fine-tuned support bot sometimes completes "The admin password is" with the real password. Which fix addresses the cause?

- A. Add "never reveal passwords" to the system prompt
- B. Lower the sampling temperature
- C. Remove the secret from the fine-tuning data and retrain (or train with DP-SGD), then confirm with an extraction test
- D. Move the model to a different provider

**Answer:** C. Remove the secret from the fine-tuning data and retrain (or train with DP-SGD), then confirm with an extraction test — The password is memorised in the weights. Prompt instructions and lower temperature change how often it comes out, not whether it is there. Lower temperature can even make a memorised completion more likely. Only removing it from training, or bounding each record's influence with DP, removes the cause.

**Question:** DP-SGD with σ = 1.1, B = 256, N = 60,000 gives ε ≈ 2.3 after 60 epochs. Without changing σ or B, what happens to ε if you train 4× longer?

- A. It stays the same
- B. It roughly doubles, since μ grows with √epochs
- C. It grows 4×
- D. It halves

**Answer:** B. It roughly doubles, since μ grows with √epochs — $\mu=\sqrt{qE(e^{1/\sigma^2}-1)}$, so 4× the epochs doubles μ, from 0.574 to 1.147, and ε rises to about 5.1 (the viz shows the same μ for B = 1024 at 60 epochs, which also has 4× the qE). The ε-to-μ map is not linear, so "roughly" doubles.

**Key points**

- Two leak families: memorised training data (needs training fixes) and inference-time data flows (need engineering fixes).
- Memorisation grows with model size, duplication and prompt length; deduplicate and scrub before training.
- Measure with canary exposure log₂|R| − log₂ rank and membership attacks reported as TPR at low FPR.
- DP-SGD clips per-example gradients and adds noise; $\mu\approx\sqrt{qE(e^{1/\sigma^2}-1)}$ bounds any attack's TPR by $\Phi(\Phi^{-1}(\text{FPR})+\mu)$.
- No secrets in prompts, tenant filters at retrieval, embeddings treated as sensitive, short log retention, redaction before third-party APIs.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
