# Generative AI in healthcare

> Healthcare LLMs mostly draft notes, letters and patient replies. Chart grounding, urgency routing tuned for sensitivity, clinician sign-off and PHI controls make that safe; exam scores (Med-PaLM 2: 86.5% on MedQA) do not.

Source: https://www.mlfoundry.in/learn/generative-ai-in-healthcare  
Section: Generative AI › Applications  
Level: core · 8 min read · updated 2026-10-21

**Generative AI in healthcare:** is the use of generative models, mostly large language models and speech-to-text, to draft clinical and administrative text for a clinician or staff member to review. The work covers visit notes, patient-message replies, discharge summaries, referral and prior-authorisation letters, and answers to questions about guidelines. The field's reference results are Med-PaLM (Singhal et al., "Large language models encode clinical knowledge", *Nature* 2023) and Med-PaLM 2 (Singhal et al., 2023). Both scored USMLE-style exam questions; neither measured patient outcomes.

The problem is clinician time. Documentation and inbox work take a large share of a physician's day, and that work is mostly text: summarising a visit, answering refill and results questions, writing letters. Generative models are good at drafting such text. They are not reliable enough to make clinical decisions alone, so the design question is always the same: who reads the output before it affects a patient?

In production systems the model sits inside the electronic health record (EHR) workflow, not in a chat window. It gets its facts from the patient's chart by retrieval (see **RAG architecture**). It writes into a draft field, and a clinician edits and signs. A separate classifier decides which inputs the model should not touch at all.

## Where it is used

- **Ambient documentation:** Speech recognition (see **Whisper**) plus **Speaker diarization** turn the visit audio into a transcript; an LLM drafts the note sections. The clinician reviews and signs. Patient consent and audio-retention rules apply.
- **Patient-message drafts:** Replies to portal messages about refills, results and logistics, grounded in the chart and sent only after clinician edit and sign-off.
- **Summaries and letters:** Discharge summaries, referral letters, prior-authorisation letters and plain-language versions of notes (see **Document summarization**).
- **Guideline and coding questions:** Retrieval over approved guidelines and formularies, with citations, for clinicians and coders. Answers are references to read, not orders.

## Risk depends on who acts on the output

A working risk tiering used by many deployment reviews: the closer the output is to a patient decision without a professional in between, the more evidence and control it needs

| Use | Who reads it first | Typical failure | Minimum controls |
| --- | --- | --- | --- |
| Admin letter, scheduling text | Staff | Wrong date or code | Spot-check sample, templates |
| Visit note draft | Signing clinician | Invented exam finding, omitted symptom | Review before signing, source transcript kept for checking |
| Patient reply draft | Clinician, then patient | Unsafe advice, missed urgency | Urgency routing, grounding, sign-off, AI-assisted label |
| Diagnosis or treatment suggestion | Clinician acting on it | Plausible but wrong recommendation | May be a regulated medical device; prospective validation |
| Patient-facing advice, no human | Patient | Harm with no one in the loop | Generally out of scope without regulatory clearance |

> Figure: A patient-message drafting pipeline. Messages cross a PHI boundary (a vendor covered by a business associate agreement, minimum necessary fields). An urgency classifier sends message A (chest tightness, score 0.93) to a nurse with no AI draft. Message B (refill transfer, score 0.04) gets a draft grounded in the medication list, with chart citations. Automatic checks block new doses or diagnoses. A clinician edits and signs, and monitoring tracks edit rates and audits missed urgent messages.

## Routing: sensitivity first, then workload

The most important control is the router. It is a classifier, often a small fine-tuned model or an LLM prompted for a score. A missed urgent message (a false negative) can harm a patient, while a false alarm costs nurse time. So the threshold is set for a target **sensitivity** on labelled validation data, and the cost shows up as workload. Base rates make this costly. Urgent messages are rare, so most flags are false alarms (see **Precision, recall & ROC**):

$$
\text{sens}=\frac{TP}{TP+FN},\qquad \text{spec}=\frac{TN}{TN+FP},\qquad \text{PPV}=\frac{\text{sens}\cdot\pi}{\text{sens}\cdot\pi+(1-\text{spec})(1-\pi)}
$$

- $\pi$: prevalence: the share of messages that are truly urgent
- $\text{PPV}$: positive predictive value: the chance that a flagged message is really urgent
- $TP,FN,FP,TN$: urgent caught, urgent missed, routine flagged, routine passed to drafting

The figure uses the binormal model: routine scores follow $\mathcal N(0,1)$ and urgent scores follow $\mathcal N(d',1)$. Then $\text{sens}=1-\Phi(t-d')$, $1-\text{spec}=1-\Phi(t)$, and $\text{AUC}=\Phi(d'/\sqrt2)$. The threshold for 99% sensitivity is $t=d'-2.326$.

> Figure: Defaults (π = 5%, d′ = 2, AUC 0.921, t = 1.00): sensitivity 84.1%, specificity 84.1%, so 7.9 urgent messages per 1,000 are missed. 19.3% go to the nurse queue, and only 21.8% of flags are truly urgent. 'Threshold for 99% sensitivity' moves t to −0.33: 0.5 missed per 1,000, but 64.6% of all messages are flagged and PPV falls to 7.7%. With a better classifier (d′ = 3), the same target needs t = 0.67 and flags 28.7%. Lower π to 1% at t = 1 and PPV drops to 5.1%.

Worked example per 1,000 messages, π = 5% (50 urgent). Values from the binormal model, computed in Python

| Setting | t | Sens. | Spec. | Missed (FN) | False alarms (FP) | PPV | Flagged |
| --- | --- | --- | --- | --- | --- | --- | --- |
| d′ = 2, default | 1.00 | 84.1% | 84.1% | 7.9 | 150.7 | 21.8% | 19.3% |
| d′ = 2, 99% sensitivity | −0.33 | 99.0% | 37.2% | 0.5 | 596.5 | 7.7% | 64.6% |
| d′ = 3, 99% sensitivity | 0.67 | 99.0% | 75.0% | 0.5 | 237.8 | 17.2% | 28.7% |

Reaching 99% sensitivity with the weaker classifier sends almost two thirds of the inbox to nurses. That removes most of the time saved. A better classifier or a narrower scope (say, only refill requests) is what makes the system pay off. A lower threshold does not.

```python
import numpy as np

def threshold_for_sensitivity(scores, labels, target=0.99):
    """Highest threshold t whose validation sensitivity is at least `target`."""
    pos = np.sort(scores[labels == 1])
    k = int(np.floor((1 - target) * len(pos)))  # urgent messages we may miss
    return pos[k]                               # flag when score >= t

rng = np.random.default_rng(0)
n_urgent, n_routine = 400, 7600                 # 5% urgent, like the figure
scores = np.r_[rng.normal(2.0, 1, n_urgent), rng.normal(0.0, 1, n_routine)]
labels = np.r_[np.ones(n_urgent, int), np.zeros(n_routine, int)]

t = threshold_for_sensitivity(scores, labels)
flag = scores >= t
print(f"t = {t:.2f}")                                     # t = -0.37
print(f"sensitivity {flag[labels == 1].mean():.3f}")      # 0.990 (4 of 400 missed)
print(f"flagged {flag.mean():.1%}, PPV {labels[flag].mean():.3f}")  # 66.0%, 0.075
```

Choosing the router threshold on a labelled validation set for a sensitivity target (numpy, runs as is; synthetic scores with d′ = 2 and 5% urgent)

> **99% on 400 cases is not 99%:** Catching 396 of 400 urgent validation messages gives a two-sided 95% Clopper–Pearson lower bound on sensitivity of 97.5%. Even 400 of 400 only shows sensitivity above 99.1%. Size the urgent validation set for the bound you need. Then re-check it on live traffic: case mix, message templates and the patient population all drift.

## What the evidence shows

**Published results (each in its own setting)**

- **Med-PaLM (2023):** Flan-PaLM 540B with prompting scored 67.6% on MedQA (USMLE-style, 4 options), the first model past the roughly 60% pass mark. Clinician raters still found gaps, which led to instruction prompt tuning in Med-PaLM.
- **Med-PaLM 2 (2023):** 86.5% on MedQA. In pairwise ratings on consumer medical questions, physicians preferred its long-form answers to physicians' answers on most quality axes.
- **GPT-4 (Nori et al., 2023):** Exceeded the USMLE passing score by over 20 points with no medical fine-tuning or special prompting.
- **Ayers et al., JAMA Intern. Med. 2023:** 195 public r/AskDocs questions: evaluators preferred ChatGPT's replies to physicians' in 78.6% of 585 evaluations, and rated them higher for quality and empathy. The physician replies were short forum posts, not clinic messages.
- **What they do not show:** Multiple-choice vignettes give the model a clean summary of the case. Real use requires finding facts in a messy chart, knowing what is missing, and outcome evidence from prospective studies.

Treat benchmark scores as a screening test for a model, not as validation of a use case. Validation means your own data and your own task. Clinicians rate the drafts for errors and omissions (see **Human evaluation and arena-style ranking**), claims are checked against the source chart (see **Hallucination and groundedness evaluation**), and edit rates and incident reports are watched after launch.

## Failure modes specific to clinical text

**Controls, in the order a request meets them**

1. **PHI boundary.** In the US, HIPAA covers protected health information; a vendor processing it for a covered entity needs a business associate agreement (BAA). In the EU, health data is a special category under GDPR Article 9. Send the minimum necessary fields, and check whether the vendor retains prompts or uses them for training (see **Privacy and data leakage**).
2. **Scope and routing.** Decide which inputs never get a draft (chest pain, suicidal ideation, new neurological symptoms) and route them by a classifier tuned for sensitivity, plus keyword rules as a backstop.
3. **Grounding.** Retrieve the medication list, allergies, problems and recent results. Instruct the model to cite chart rows, and reject drafts with uncited clinical claims. Hallucinated findings, such as a normal exam that was never done, are the most cited failure of ambient notes (see **Hallucination**).
4. **Hard checks.** Run deterministic checks for doses against a formulary, drug–allergy conflicts, and no new diagnosis or dose change in a patient reply. Use **Structured output (JSON mode)** to make the fields checkable.
5. **Human sign-off.** The clinician sees the draft and its sources and edits, signs or discards it. Label the reply as AI-assisted where policy or law requires.
6. **Monitoring.** Track edit distance between draft and sent text, the discard rate, sampled audits of routed and unrouted messages, and differences across patient groups (see **Bias and fairness in generative models**).

> **Automation bias grows as drafts get better:** If 99 drafts in a row are fine, reviewers stop reading the 100th. Plan for this. Show the source next to every clinical claim, highlight changed values such as doses, dates and lab numbers, and sample signed drafts for audit. The human in the loop is a control only while the human actually checks.

> **Regulation, in general terms (not legal advice):** In the US, software that informs clinical decisions can be a medical device regulated by the FDA. The 21st Century Cures Act (2016) and the FDA's 2022 clinical decision support guidance exclude some tools, for example those that let a professional independently review the basis of a recommendation. In the EU, AI that is a medical device needing third-party assessment under the MDR is high-risk under the AI Act (Regulation (EU) 2024/1689). Rules and guidance change; check the current versions with counsel and your regulatory team.

**Strengths**

- Drafting cuts time on documentation and inbox work while the clinician keeps the decision.
- Chart grounding plus citations makes each claim checkable in seconds.
- Plain-language rewrites and translations make notes easier for patients to read.

**Limitations**

- Errors are fluent and plausible; an omitted symptom is harder to spot than a wrong one.
- High-sensitivity routing can flag most of the inbox and wipe out the time saved.
- Review fatigue turns sign-off into a rubber stamp.
- Validation, PHI contracts and regulatory review cost more than the model.

**Question:** A router has sensitivity 95% and specificity 90% for urgent messages, which make up 2% of the inbox. Roughly what share of flagged messages are truly urgent?

- A. About 95%
- B. About 90%
- C. About 50%
- D. About 16%

**Answer:** D. About 16% — $\text{PPV}=\frac{0.95\cdot0.02}{0.95\cdot0.02+0.10\cdot0.98}=\frac{0.019}{0.019+0.098}=0.162$. Most flags are false alarms because routine messages outnumber urgent ones 49 to 1.

**Question:** Which control most directly limits harm from an LLM inventing a normal physical exam in an ambient visit note?

- A. Raising the sampling temperature
- B. Requiring each finding to cite the transcript, with clinician review before signing
- C. A larger model with a higher MedQA score
- D. Removing PHI from the prompt

**Answer:** B. Requiring each finding to cite the transcript, with clinician review before signing — The failure is an unsupported claim. Grounding each finding in the transcript makes it checkable, and sign-off puts a professional between the draft and the record. Benchmark scores do not measure this failure, and de-identification addresses privacy, not accuracy.

**Key points**

- Applied healthcare GenAI is mostly drafting (notes, replies, letters, summaries) for a professional to review; risk rises the closer output gets to a patient decision without a human.
- Route high-risk inputs away from the model with a classifier tuned for sensitivity; PPV = sens·π / (sens·π + (1 − spec)(1 − π)) shows why most flags are false alarms.
- In the figure, 99% sensitivity at d′ = 2 flags 64.6% of the inbox; d′ = 3 needs only 28.7%. Classifier quality pays, lowering the threshold does not.
- Ground drafts in the chart with citations, run deterministic dose and allergy checks, require sign-off, and monitor edit rates and missed cases.
- Exam results (Flan-PaLM 67.6%, Med-PaLM 2 86.5% on MedQA) are screening evidence, not validation of a workflow.
- HIPAA/BAA, GDPR Article 9, FDA device rules and the EU AI Act shape deployment; confirm the current rules with counsel.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
