Privacy and data leakage

Generative AI leaks private data two ways: memorised training text recovered by extraction or membership attacks (Carlini et al. 2021), and prompts, RAG indexes and logs at inference; DP-SGD bounds the first by (ε, δ), access control the second.

8 min read744 wordscore0 viewsupdated 2026-10-21

Definition

Privacy and data leakage in generative AI means private information reaching someone who should not see it, through the model or the system around it. Two families matter. Training-data leakage: the model memorises parts of its training set and reproduces them, shown at scale by Carlini et al. ("Extracting Training Data from Large Language Models", USENIX Security 2021). Inference-time leakage: prompts, system prompts, retrieved documents, embeddings and logs flow to the wrong user or a third party. OWASP's Top 10 for LLM Applications (2025) lists these as LLM02 Sensitive Information Disclosure, LLM07 System Prompt Leakage and LLM08 Vector and Embedding Weaknesses.

The two families need different fixes. Memorisation is a property of training: it is set by the data, its duplication and the optimiser, and only changes to training remove it. Inference-time leakage needs no memorisation at all. It comes from how the application moves data, so it is fixed with ordinary engineering: redaction, access control, retention limits and contracts.

Memorisation and extraction

Carlini et al. (2021) sampled 600,000 generations from GPT-2, ranked them by how much more likely GPT-2 found them than a reference model or zlib compression did, and checked the top 1,800 against the training set. 604 were verbatim training text, including names, phone numbers, email addresses and code. Some of these strings appeared in only one training document.

Follow-up work measured what drives it. Carlini et al. ("Quantifying Memorization Across Neural Language Models", ICLR 2023) found extractable memorisation grows log-linearly with model size, with how often a string is duplicated, and with the length of the prompt prefix. Kandpal, Wallace and Raffel (ICML 2022) showed deduplicating the training set sharply reduces regenerated training text. Nasr et al. (2023) extracted training data from the production ChatGPT (gpt-3.5-turbo) with a "divergence" prompt asking it to repeat one word forever, after which it drifted into memorised text.

1. training-time leakage: the data ends up in the weightstraining corpus...mail ana@corp.example...canary: my PIN is 7302same page copied ×40(dupes are memorised most)pre-train / tuneminimise loss on Drare + repeatedstrings getmemorised verbatimreleased model / APIextraction:"my PIN is" → "7302"membership inference:loss(x) low → x ∈ D?✓ dedup, scrub PII, plant canaries✓ DP-SGD: clip each gradient, add noise✓ block outputs that match training text2. inference-time leakage: no training involveduser prompt"Draft a reply toJ. Smith, DOB 1984-03-02,acct 12-3456-78 ..."PII leaves your boundaryLLM appsystem prompt: key?RAG index A | Bembeddings storedlogs + tracesmodel providerprompts + outputs passthrough a third partyretention period?used for training?leaks:system-prompt extraction · tenant B's chunk retrieved for tenant Atext rebuilt from stored embeddings · PII sitting in logs✓ redact or pseudonymise PII before sending✓ no secrets in prompts; ACL filter at retrieval✓ zero/short retention, training opt-out, DPA✓ = mitigation at that stage. Lane 2 needs no memorisation: the app moves the data.
fig.Lane 1: a crawled email address, a planted canary ("my PIN is 7302") and a page duplicated 40 times enter training. Rare and repeated strings get memorised, so the released model completes "my PIN is" with "7302", and its low loss on a record reveals membership. Lane 2: no memorisation needed. A user pastes a date of birth and account number, the app holds a system prompt, a two-tenant RAG index, stored embeddings and logs, and everything passes through the model provider. Green ticks are the control at each stage.

Two attacks are measured. Extraction recovers a training string from the model. Membership inference (Shokri et al., IEEE S&P 2017) decides whether a given record was in the training set, usually by thresholding the model's loss on it. Report membership attacks as the true-positive rate at a low false-positive rate such as 0.1% or 1% (Carlini et al., "Membership Inference Attacks From First Principles", 2022). Average accuracy hides the few records that are exposed with high confidence.

exposure(s)=log⁡2∣R∣−log⁡2rank⁡θ(s)\text{exposure}(s)=\log_2|\mathcal R|-\log_2\operatorname{rank}_\theta(s)
ss
a random canary inserted into the training data, e.g. "my PIN is 7302"
R\mathcal R
the space of possible canary values (here 10410^4 PINs)
rank⁡θ(s)\operatorname{rank}_\theta(s)
position of the true canary when all candidates are sorted by model perplexity (1 = most likely)

Exposure comes from "The Secret Sharer" (Carlini et al., USENIX Security 2019). An exposure close to log⁡2∣R∣\log_2|\mathcal R| means the model ranks the secret first and an attacker can read it off. Planting canaries and measuring their exposure is a cheap way to audit your own fine-tuning run.

Rank of true PINlog₂ rankExposure (bits)Reading
1013.29maximum: the model's top guess is the secret
31.5811.70found within 3 guesses
1006.646.64partly memorised
5,00012.291.00no better than about half the candidates
Worked example: a 4-digit PIN canary, |R| = 10⁴, log₂ 10⁴ = 13.29 bits

Differential privacy

Pr⁡[M(D)∈S]  ≤  eε Pr⁡[M(D′)∈S]+δfor all S and all D,D′ differing in one record\Pr[M(D)\in S]\;\le\;e^{\varepsilon}\,\Pr[M(D')\in S]+\delta\quad\text{for all } S \text{ and all } D,D' \text{ differing in one record}
MM
the randomised training algorithm; its output is the model
ε\varepsilon
privacy budget: smaller means one record changes the output distribution less
δ\delta
small failure probability, chosen well below 1/N1/N

The definition is due to Dwork, McSherry, Nissim and Smith (2006). It bounds every attacker, including membership inference: whatever test is run, its true-positive and false-positive rates satisfy TPR≤eε FPR+δ\text{TPR}\le e^\varepsilon\,\text{FPR}+\delta. DP-SGD (Abadi et al., CCS 2016) makes SGD private by clipping each example's gradient and adding Gaussian noise:

g~=1B(∑i∈batchclip⁡C(gi)+N(0,σ2C2I)),μ≈qT(e1/σ2−1),TPR≤Φ ⁣(Φ−1(FPR)+μ)\tilde g=\frac1B\Big(\sum_{i\in\text{batch}}\operatorname{clip}_C(g_i)+\mathcal N(0,\sigma^2C^2I)\Big),\qquad \mu\approx q\sqrt{T\left(e^{1/\sigma^2}-1\right)},\quad \text{TPR}\le\Phi\!\left(\Phi^{-1}(\text{FPR})+\mu\right)
clip⁡C\operatorname{clip}_C
rescale gig_i to norm at most CC, so one example's influence is bounded
σ\sigma
noise multiplier
q,  Tq,\;T
sampling rate B/NB/N and number of steps
μ\mu
Gaussian-DP parameter (central-limit accountant, Bu, Dong, Long and Su 2020); it converts to an (ε,δ)(\varepsilon,\delta) pair
DP-SGD, N = 60,000, clip C = 1, δ = 10⁻⁵ (Gaussian-DP CLT accountant)best membership attack, μ = 0.574TPR 23.9% @ FPR 10%random guess000.50.511false-positive ratetrue-positive rateprivacy budget ε vs epochs2468100100200epochs (curve clipped at ε = 12)q = B/N = 0.00427 · steps T = 14,062 · noise std on mean gradient σC/B = 0.0043ε = 2.32 at δ = 10⁻⁵best attack: TPR 3.98% @ FPR 1%, 23.9% @ FPR 10%μ = q·√(T·(e^(1/σ²) − 1)); no attack can beat TPR = Φ(Φ⁻¹(FPR) + μ). No noise (σ → 0): μ → ∞, no bound.
batch B
fig.DP-SGD on N = 60,000 examples with C = 1 and δ = 10⁻⁵. Defaults (σ = 1.1, B = 256, 60 epochs): T = 14,062 steps, μ = 0.574, ε = 2.32; the best possible membership attack reaches only 3.98% TPR at 1% FPR and 23.9% at 10%. Drop σ to 0.6: ε = 9.78 and TPR 35.9% at 1% FPR. Raise σ to 2.0: ε = 1.01. Keep σ = 1.1 but train 200 epochs: ε = 4.62. B = 1024 for the same 60 epochs: ε = 5.14.

Since T=EN/BT=EN/B, the formula reduces to μ=qE (e1/σ2−1)\mu=\sqrt{qE\,(e^{1/\sigma^2}-1)} for EE epochs. Privacy cost grows with the square root of the epochs and of the batch fraction. B = 64 for 60 epochs and B = 256 for 15 epochs both give μ = 0.287 and ε = 1.08. Larger batches still help utility, because the noise on the averaged gradient is σC/B\sigma C/B, so σ is usually raised with B.

  1. 1Sampling rate: q=256/60,000=0.004267q=256/60{,}000=0.004267; steps T=⌊60⋅60,000/256⌋=14,062T=\lfloor 60\cdot60{,}000/256\rfloor=14{,}062.
  2. 2Noise term: 1/σ2=1/1.21=0.82641/\sigma^2=1/1.21=0.8264, e0.8264−1=1.2852e^{0.8264}-1=1.2852.
  3. 3T⋅1.2852=18,072=134.43\sqrt{T\cdot1.2852}=\sqrt{18{,}072}=134.43; μ=0.004267×134.43=0.574\mu=0.004267\times134.43=0.574.
  4. 4Solve δ(ε)=Φ(−ε/μ+μ/2)−eεΦ(−ε/μ−μ/2)=10−5\delta(\varepsilon)=\Phi(-\varepsilon/\mu+\mu/2)-e^{\varepsilon}\Phi(-\varepsilon/\mu-\mu/2)=10^{-5}: ε=2.32\varepsilon=2.32.
  5. 5Attack bound at FPR 1%: Φ(Φ−1(0.01)+0.574)=Φ(−2.326+0.574)=Φ(−1.752)=3.98%\Phi(\Phi^{-1}(0.01)+0.574)=\Phi(-2.326+0.574)=\Phi(-1.752)=3.98\%, against 1% for a random guess.

Use when

  • +The only defence with a proven bound that holds against every attack, including future ones.
  • +Composes: the accountant tracks the budget across epochs and multiple releases.
  • +Fine-tuning a public pre-trained model with DP-SGD on private data loses far less accuracy than training from scratch with DP.

Avoid when

  • −Per-example gradients cost memory and time; clipping and noise lower accuracy, most for rare classes and groups.
  • −Protects records, not facts: if a fact appears in many records it can still be learned and repeated.
  • −ε values used in practice (often 1–10) give weak formal guarantees at the top of that range.

Inference-time leakage

PathExampleControl
Prompts to a third partyUsers paste contracts or patient notes into a hosted modelRedact or pseudonymise first; contract for zero or short retention and no training on inputs
System-prompt leakagePrompt injection makes the bot print its instructions, including an API keyNever put secrets in prompts; assume the system prompt is public (see Prompt injection)
Cross-tenant retrievalOne shared vector index returns tenant B's chunk to tenant AFilter by tenant and user permissions at query time, not after generation
Embedding inversionvec2text (Morris et al. 2023) reconstructs input text from its embeddingTreat stored embeddings as sensitive as the text they encode
Logs and tracesFull prompts kept indefinitely in observability toolsMinimise, mask, set retention; restrict access
Leak paths that need no memorisation, and the usual control

Checkable specifics

GPT-2 extraction (2021)
600,000 samples → 1,800 candidates → 604 verbatim training sequences
Canary exposure
maximum log⁡2∣R∣\log_2|\mathcal R|; 13.29 bits for a 4-digit PIN, 29.9 bits for a 9-digit number
DP accounting
δ is set well below 1/N; DP-SGD papers commonly report ε between about 1 and 10
GDPR (EU 2016/679)
data minimisation (Art. 5(1)(c)), right to erasure (Art. 17), data protection by design (Art. 25) apply to personal data in prompts, logs and training sets
python
1import re
2
3PATTERNS = [ # order matters: most specific first
4 ("EMAIL", re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+")),
5 ("DATE", re.compile(r"\b\d{4}-\d{2}-\d{2}\b")),
6 ("ACCT", re.compile(r"\b\d{2}-\d{4}-\d{2}\b")),
7 ("PHONE", re.compile(r"\+?\d[\d ()-]{7,}\d")),
8]
9
10def pseudonymise(text):
11 """Replace PII with stable placeholders; return text and the reverse map."""
12 mapping = {}
13 for label, pat in PATTERNS:
14 def sub(m):
15 key = next((k for k, v in mapping.items() if v == m.group()), None)
16 if key is None:
17 key = f"<{label}_{sum(k.startswith('<' + label) for k in mapping) + 1}>"
18 mapping[key] = m.group()
19 return key
20 text = pat.sub(sub, text)
21 return text, mapping
22
23def restore(text, mapping):
24 for key, val in mapping.items():
25 text = text.replace(key, val)
26 return text
27
28prompt = ("Draft a reply to J. Smith (j.smith@mail.example, +44 20 7946 0958), "
29 "DOB 1984-03-02, about account 12-3456-78.")
30safe, m = pseudonymise(prompt)
31print(safe)
32# Draft a reply to J. Smith (<EMAIL_1>, <PHONE_1>), DOB <DATE_1>, about account <ACCT_1>.
33reply = "Dear J. Smith, we have updated account <ACCT_1>." # what the API returns
34print(restore(reply, m))
35# Dear J. Smith, we have updated account 12-3456-78.
Pseudonymise PII before a prompt leaves your boundary and restore it in the reply (standard library only, runs as is). Regexes miss names such as "J. Smith"; production systems add an NER-based detector

Check yourself

A fine-tuned support bot sometimes completes "The admin password is" with the real password. Which fix addresses the cause?

Check yourself

DP-SGD with σ = 1.1, B = 256, N = 60,000 gives ε ≈ 2.3 after 60 epochs. Without changing σ or B, what happens to ε if you train 4× longer?

Remember

  • ✓Two leak families: memorised training data (needs training fixes) and inference-time data flows (need engineering fixes).
  • ✓Memorisation grows with model size, duplication and prompt length; deduplicate and scrub before training.
  • ✓Measure with canary exposure log₂|R| − log₂ rank and membership attacks reported as TPR at low FPR.
  • ✓DP-SGD clips per-example gradients and adds noise; μ≈qE(e1/σ2−1)\mu\approx\sqrt{qE(e^{1/\sigma^2}-1)} bounds any attack's TPR by Φ(Φ−1(FPR)+μ)\Phi(\Phi^{-1}(\text{FPR})+\mu).
  • ✓No secrets in prompts, tenant filters at retrieval, embeddings treated as sensitive, short log retention, redaction before third-party APIs.
← back to Generative AI roadmap