Definition
Privacy and data leakage in generative AI means private information reaching someone who should not see it, through the model or the system around it. Two families matter. Training-data leakage: the model memorises parts of its training set and reproduces them, shown at scale by Carlini et al. ("Extracting Training Data from Large Language Models", USENIX Security 2021). Inference-time leakage: prompts, system prompts, retrieved documents, embeddings and logs flow to the wrong user or a third party. OWASP's Top 10 for LLM Applications (2025) lists these as LLM02 Sensitive Information Disclosure, LLM07 System Prompt Leakage and LLM08 Vector and Embedding Weaknesses.
The two families need different fixes. Memorisation is a property of training: it is set by the data, its duplication and the optimiser, and only changes to training remove it. Inference-time leakage needs no memorisation at all. It comes from how the application moves data, so it is fixed with ordinary engineering: redaction, access control, retention limits and contracts.
Memorisation and extraction
Carlini et al. (2021) sampled 600,000 generations from GPT-2, ranked them by how much more likely GPT-2 found them than a reference model or zlib compression did, and checked the top 1,800 against the training set. 604 were verbatim training text, including names, phone numbers, email addresses and code. Some of these strings appeared in only one training document.
Follow-up work measured what drives it. Carlini et al. ("Quantifying Memorization Across Neural Language Models", ICLR 2023) found extractable memorisation grows log-linearly with model size, with how often a string is duplicated, and with the length of the prompt prefix. Kandpal, Wallace and Raffel (ICML 2022) showed deduplicating the training set sharply reduces regenerated training text. Nasr et al. (2023) extracted training data from the production ChatGPT (gpt-3.5-turbo) with a "divergence" prompt asking it to repeat one word forever, after which it drifted into memorised text.
Two attacks are measured. Extraction recovers a training string from the model. Membership inference (Shokri et al., IEEE S&P 2017) decides whether a given record was in the training set, usually by thresholding the model's loss on it. Report membership attacks as the true-positive rate at a low false-positive rate such as 0.1% or 1% (Carlini et al., "Membership Inference Attacks From First Principles", 2022). Average accuracy hides the few records that are exposed with high confidence.
- a random canary inserted into the training data, e.g. "my PIN is 7302"
- the space of possible canary values (here PINs)
- position of the true canary when all candidates are sorted by model perplexity (1 = most likely)
Exposure comes from "The Secret Sharer" (Carlini et al., USENIX Security 2019). An exposure close to means the model ranks the secret first and an attacker can read it off. Planting canaries and measuring their exposure is a cheap way to audit your own fine-tuning run.
| Rank of true PIN | log₂ rank | Exposure (bits) | Reading |
|---|---|---|---|
| 1 | 0 | 13.29 | maximum: the model's top guess is the secret |
| 3 | 1.58 | 11.70 | found within 3 guesses |
| 100 | 6.64 | 6.64 | partly memorised |
| 5,000 | 12.29 | 1.00 | no better than about half the candidates |
Differential privacy
- the randomised training algorithm; its output is the model
- privacy budget: smaller means one record changes the output distribution less
- small failure probability, chosen well below
The definition is due to Dwork, McSherry, Nissim and Smith (2006). It bounds every attacker, including membership inference: whatever test is run, its true-positive and false-positive rates satisfy . DP-SGD (Abadi et al., CCS 2016) makes SGD private by clipping each example's gradient and adding Gaussian noise:
- rescale to norm at most , so one example's influence is bounded
- noise multiplier
- sampling rate and number of steps
- Gaussian-DP parameter (central-limit accountant, Bu, Dong, Long and Su 2020); it converts to an pair
Since , the formula reduces to for epochs. Privacy cost grows with the square root of the epochs and of the batch fraction. B = 64 for 60 epochs and B = 256 for 15 epochs both give μ = 0.287 and ε = 1.08. Larger batches still help utility, because the noise on the averaged gradient is , so σ is usually raised with B.
- 1Sampling rate: ; steps .
- 2Noise term: , .
- 3; .
- 4Solve : .
- 5Attack bound at FPR 1%: , against 1% for a random guess.
Use when
- +The only defence with a proven bound that holds against every attack, including future ones.
- +Composes: the accountant tracks the budget across epochs and multiple releases.
- +Fine-tuning a public pre-trained model with DP-SGD on private data loses far less accuracy than training from scratch with DP.
Avoid when
- −Per-example gradients cost memory and time; clipping and noise lower accuracy, most for rare classes and groups.
- −Protects records, not facts: if a fact appears in many records it can still be learned and repeated.
- −ε values used in practice (often 1–10) give weak formal guarantees at the top of that range.
Inference-time leakage
| Path | Example | Control |
|---|---|---|
| Prompts to a third party | Users paste contracts or patient notes into a hosted model | Redact or pseudonymise first; contract for zero or short retention and no training on inputs |
| System-prompt leakage | Prompt injection makes the bot print its instructions, including an API key | Never put secrets in prompts; assume the system prompt is public (see Prompt injection) |
| Cross-tenant retrieval | One shared vector index returns tenant B's chunk to tenant A | Filter by tenant and user permissions at query time, not after generation |
| Embedding inversion | vec2text (Morris et al. 2023) reconstructs input text from its embedding | Treat stored embeddings as sensitive as the text they encode |
| Logs and traces | Full prompts kept indefinitely in observability tools | Minimise, mask, set retention; restrict access |
Checkable specifics
- GPT-2 extraction (2021)
- 600,000 samples → 1,800 candidates → 604 verbatim training sequences
- Canary exposure
- maximum ; 13.29 bits for a 4-digit PIN, 29.9 bits for a 9-digit number
- DP accounting
- δ is set well below 1/N; DP-SGD papers commonly report ε between about 1 and 10
- GDPR (EU 2016/679)
- data minimisation (Art. 5(1)(c)), right to erasure (Art. 17), data protection by design (Art. 25) apply to personal data in prompts, logs and training sets
1import re23PATTERNS = [ # order matters: most specific first4 ("EMAIL", re.compile(r"[\w.+-]+@[\w-]+(?:\.[\w-]+)+")),5 ("DATE", re.compile(r"\b\d{4}-\d{2}-\d{2}\b")),6 ("ACCT", re.compile(r"\b\d{2}-\d{4}-\d{2}\b")),7 ("PHONE", re.compile(r"\+?\d[\d ()-]{7,}\d")),8]910def pseudonymise(text):11 """Replace PII with stable placeholders; return text and the reverse map."""12 mapping = {}13 for label, pat in PATTERNS:14 def sub(m):15 key = next((k for k, v in mapping.items() if v == m.group()), None)16 if key is None:17 key = f"<{label}_{sum(k.startswith('<' + label) for k in mapping) + 1}>"18 mapping[key] = m.group()19 return key20 text = pat.sub(sub, text)21 return text, mapping2223def restore(text, mapping):24 for key, val in mapping.items():25 text = text.replace(key, val)26 return text2728prompt = ("Draft a reply to J. Smith (j.smith@mail.example, +44 20 7946 0958), "29 "DOB 1984-03-02, about account 12-3456-78.")30safe, m = pseudonymise(prompt)31print(safe)32# Draft a reply to J. Smith (<EMAIL_1>, <PHONE_1>), DOB <DATE_1>, about account <ACCT_1>.33reply = "Dear J. Smith, we have updated account <ACCT_1>." # what the API returns34print(restore(reply, m))35# Dear J. Smith, we have updated account 12-3456-78.Check yourself
A fine-tuned support bot sometimes completes "The admin password is" with the real password. Which fix addresses the cause?
Check yourself
DP-SGD with σ = 1.1, B = 256, N = 60,000 gives ε ≈ 2.3 after 60 epochs. Without changing σ or B, what happens to ε if you train 4× longer?
Remember
- ✓Two leak families: memorised training data (needs training fixes) and inference-time data flows (need engineering fixes).
- ✓Memorisation grows with model size, duplication and prompt length; deduplicate and scrub before training.
- ✓Measure with canary exposure log₂|R| − log₂ rank and membership attacks reported as TPR at low FPR.
- ✓DP-SGD clips per-example gradients and adds noise; bounds any attack's TPR by .
- ✓No secrets in prompts, tenant filters at retrieval, embeddings treated as sensitive, short log retention, redaction before third-party APIs.