Generative AI in healthcare

Healthcare LLMs mostly draft notes, letters and patient replies. Chart grounding, urgency routing tuned for sensitivity, clinician sign-off and PHI controls make that safe; exam scores (Med-PaLM 2: 86.5% on MedQA) do not.

8 min read993 wordscore0 viewsupdated 2026-10-21

Definition

Generative AI in healthcare is the use of generative models, mostly large language models and speech-to-text, to draft clinical and administrative text for a clinician or staff member to review. The work covers visit notes, patient-message replies, discharge summaries, referral and prior-authorisation letters, and answers to questions about guidelines. The field's reference results are Med-PaLM (Singhal et al., "Large language models encode clinical knowledge", Nature 2023) and Med-PaLM 2 (Singhal et al., 2023). Both scored USMLE-style exam questions; neither measured patient outcomes.

The problem is clinician time. Documentation and inbox work take a large share of a physician's day, and that work is mostly text: summarising a visit, answering refill and results questions, writing letters. Generative models are good at drafting such text. They are not reliable enough to make clinical decisions alone, so the design question is always the same: who reads the output before it affects a patient?

In production systems the model sits inside the electronic health record (EHR) workflow, not in a chat window. It gets its facts from the patient's chart by retrieval (see RAG architecture). It writes into a draft field, and a clinician edits and signs. A separate classifier decides which inputs the model should not touch at all.

Where it is used

Where it's used

Ambient documentation

Speech recognition (see Whisper) plus Speaker diarization turn the visit audio into a transcript; an LLM drafts the note sections. The clinician reviews and signs. Patient consent and audio-retention rules apply.

Patient-message drafts

Replies to portal messages about refills, results and logistics, grounded in the chart and sent only after clinician edit and sign-off.

Summaries and letters

Discharge summaries, referral letters, prior-authorisation letters and plain-language versions of notes (see Document summarization).

Guideline and coding questions

Retrieval over approved guidelines and formularies, with citations, for clinicians and coders. Answers are references to read, not orders.

Risk depends on who acts on the output

UseWho reads it firstTypical failureMinimum controls
Admin letter, scheduling textStaffWrong date or codeSpot-check sample, templates
Visit note draftSigning clinicianInvented exam finding, omitted symptomReview before signing, source transcript kept for checking
Patient reply draftClinician, then patientUnsafe advice, missed urgencyUrgency routing, grounding, sign-off, AI-assisted label
Diagnosis or treatment suggestionClinician acting on itPlausible but wrong recommendationMay be a regulated medical device; prospective validation
Patient-facing advice, no humanPatientHarm with no one in the loopGenerally out of scope without regulatory clearance
A working risk tiering used by many deployment reviews: the closer the output is to a patient decision without a professional in between, the more evidence and control it needs
patient-message drafting: the model drafts, a clinician decidesportal inboxA: chest tightness since7am, take extra metoprolol?B: please move my lisinoprilrefill to Elm St Pharmacyfree text, may contain PHI1 PHI boundaryBAA-covered endpoint,minimum necessary data2 urgency classifierA: 0.93 · B: 0.04route if score ≥ t = 0.30A → nurse queueno AI draft is shownnurse calls back(local policy: < 15 min)3 grounded draft (B)retrieve: med list,pharmacy, refills leftprompt: cite chart rows4 automatic checksevery claim cites chartno new dose or diagnosisreading level ≤ grade 85 clinician reviewedit, sign, sendor discard and rewritereply labelled AI-assisted6 monitoringedit rate per draftaudit sample of missesre-check t on new data≥ t< tpasses checksdraft for B, shown to the clinician:Hi Ms. R., your lisinopril 10 mg refill(2 refills left) [chart: meds #3] has beenmoved to Elm St Pharmacy [chart: pharmacy].Call us if you notice dizziness or swelling.t is set on labelled validation messages for a target sensitivity, not by the LLM itself
fig.A patient-message drafting pipeline. Messages cross a PHI boundary (a vendor covered by a business associate agreement, minimum necessary fields). An urgency classifier sends message A (chest tightness, score 0.93) to a nurse with no AI draft. Message B (refill transfer, score 0.04) gets a draft grounded in the medication list, with chart citations. Automatic checks block new doses or diagnoses. A clinician edits and signs, and monitoring tracks edit rates and audits missed urgent messages.

Routing: sensitivity first, then workload

The most important control is the router. It is a classifier, often a small fine-tuned model or an LLM prompted for a score. A missed urgent message (a false negative) can harm a patient, while a false alarm costs nurse time. So the threshold is set for a target sensitivity on labelled validation data, and the cost shows up as workload. Base rates make this costly. Urgent messages are rare, so most flags are false alarms (see Precision, recall & ROC):

sens=TPTP+FN,spec=TNTN+FP,PPV=sens⋅πsens⋅π+(1−spec)(1−π)\text{sens}=\frac{TP}{TP+FN},\qquad \text{spec}=\frac{TN}{TN+FP},\qquad \text{PPV}=\frac{\text{sens}\cdot\pi}{\text{sens}\cdot\pi+(1-\text{spec})(1-\pi)}
π\pi
prevalence: the share of messages that are truly urgent
PPV\text{PPV}
positive predictive value: the chance that a flagged message is really urgent
TP,FN,FP,TNTP,FN,FP,TN
urgent caught, urgent missed, routine flagged, routine passed to drafting

The figure uses the binormal model: routine scores follow N(0,1)\mathcal N(0,1) and urgent scores follow N(d′,1)\mathcal N(d',1). Then sens=1−Φ(t−d′)\text{sens}=1-\Phi(t-d'), 1−spec=1−Φ(t)1-\text{spec}=1-\Phi(t), and AUC=Φ(d′/2)\text{AUC}=\Phi(d'/\sqrt2). The threshold for 99% sensitivity is t=d′−2.326t=d'-2.326.

score ≥ t → nurse queue (no AI draft) · score < t → AI draft + clinician sign-offt = 1.00routine N(0,1)urgent N(2.0,1)classifier score density (each class normalised)ROC, AUC = 0.9211 − specificity10per 1,000 messages (5.0% urgent)flaggednot flaggedurgentroutine42.17.9150.7799.3TP urgent caught · FN urgent missed · FP routine sent to nurse · TN routine draftedsensitivity84.1%specificity84.1%PPV of a flag21.8%NPV99.02%missed / 1,0007.9to nurse queue19.3%PPV = sens·π / (sens·π + (1 − spec)(1 − π)): rare urgent messages make most flags false alarms
presets
fig.Defaults (π = 5%, d′ = 2, AUC 0.921, t = 1.00): sensitivity 84.1%, specificity 84.1%, so 7.9 urgent messages per 1,000 are missed. 19.3% go to the nurse queue, and only 21.8% of flags are truly urgent. 'Threshold for 99% sensitivity' moves t to −0.33: 0.5 missed per 1,000, but 64.6% of all messages are flagged and PPV falls to 7.7%. With a better classifier (d′ = 3), the same target needs t = 0.67 and flags 28.7%. Lower π to 1% at t = 1 and PPV drops to 5.1%.
SettingtSens.Spec.Missed (FN)False alarms (FP)PPVFlagged
d′ = 2, default1.0084.1%84.1%7.9150.721.8%19.3%
d′ = 2, 99% sensitivity−0.3399.0%37.2%0.5596.57.7%64.6%
d′ = 3, 99% sensitivity0.6799.0%75.0%0.5237.817.2%28.7%
Worked example per 1,000 messages, π = 5% (50 urgent). Values from the binormal model, computed in Python

Reaching 99% sensitivity with the weaker classifier sends almost two thirds of the inbox to nurses. That removes most of the time saved. A better classifier or a narrower scope (say, only refill requests) is what makes the system pay off. A lower threshold does not.

python
1import numpy as np
2
3def threshold_for_sensitivity(scores, labels, target=0.99):
4 """Highest threshold t whose validation sensitivity is at least `target`."""
5 pos = np.sort(scores[labels == 1])
6 k = int(np.floor((1 - target) * len(pos))) # urgent messages we may miss
7 return pos[k] # flag when score >= t
8
9rng = np.random.default_rng(0)
10n_urgent, n_routine = 400, 7600 # 5% urgent, like the figure
11scores = np.r_[rng.normal(2.0, 1, n_urgent), rng.normal(0.0, 1, n_routine)]
12labels = np.r_[np.ones(n_urgent, int), np.zeros(n_routine, int)]
13
14t = threshold_for_sensitivity(scores, labels)
15flag = scores >= t
16print(f"t = {t:.2f}") # t = -0.37
17print(f"sensitivity {flag[labels == 1].mean():.3f}") # 0.990 (4 of 400 missed)
18print(f"flagged {flag.mean():.1%}, PPV {labels[flag].mean():.3f}") # 66.0%, 0.075
Choosing the router threshold on a labelled validation set for a sensitivity target (numpy, runs as is; synthetic scores with d′ = 2 and 5% urgent)

What the evidence shows

Published results (each in its own setting)

Med-PaLM (2023)
Flan-PaLM 540B with prompting scored 67.6% on MedQA (USMLE-style, 4 options), the first model past the roughly 60% pass mark. Clinician raters still found gaps, which led to instruction prompt tuning in Med-PaLM.
Med-PaLM 2 (2023)
86.5% on MedQA. In pairwise ratings on consumer medical questions, physicians preferred its long-form answers to physicians' answers on most quality axes.
GPT-4 (Nori et al., 2023)
Exceeded the USMLE passing score by over 20 points with no medical fine-tuning or special prompting.
Ayers et al., JAMA Intern. Med. 2023
195 public r/AskDocs questions: evaluators preferred ChatGPT's replies to physicians' in 78.6% of 585 evaluations, and rated them higher for quality and empathy. The physician replies were short forum posts, not clinic messages.
What they do not show
Multiple-choice vignettes give the model a clean summary of the case. Real use requires finding facts in a messy chart, knowing what is missing, and outcome evidence from prospective studies.

Treat benchmark scores as a screening test for a model, not as validation of a use case. Validation means your own data and your own task. Clinicians rate the drafts for errors and omissions (see Human evaluation and arena-style ranking), claims are checked against the source chart (see Hallucination and groundedness evaluation), and edit rates and incident reports are watched after launch.

Failure modes specific to clinical text

Controls, in the order a request meets them

  1. 1PHI boundary. In the US, HIPAA covers protected health information; a vendor processing it for a covered entity needs a business associate agreement (BAA). In the EU, health data is a special category under GDPR Article 9. Send the minimum necessary fields, and check whether the vendor retains prompts or uses them for training (see Privacy and data leakage).
  2. 2Scope and routing. Decide which inputs never get a draft (chest pain, suicidal ideation, new neurological symptoms) and route them by a classifier tuned for sensitivity, plus keyword rules as a backstop.
  3. 3Grounding. Retrieve the medication list, allergies, problems and recent results. Instruct the model to cite chart rows, and reject drafts with uncited clinical claims. Hallucinated findings, such as a normal exam that was never done, are the most cited failure of ambient notes (see Hallucination).
  4. 4Hard checks. Run deterministic checks for doses against a formulary, drug–allergy conflicts, and no new diagnosis or dose change in a patient reply. Use Structured output (JSON mode) to make the fields checkable.
  5. 5Human sign-off. The clinician sees the draft and its sources and edits, signs or discards it. Label the reply as AI-assisted where policy or law requires.
  6. 6Monitoring. Track edit distance between draft and sent text, the discard rate, sampled audits of routed and unrouted messages, and differences across patient groups (see Bias and fairness in generative models).

Use when

  • +Drafting cuts time on documentation and inbox work while the clinician keeps the decision.
  • +Chart grounding plus citations makes each claim checkable in seconds.
  • +Plain-language rewrites and translations make notes easier for patients to read.

Avoid when

  • −Errors are fluent and plausible; an omitted symptom is harder to spot than a wrong one.
  • −High-sensitivity routing can flag most of the inbox and wipe out the time saved.
  • −Review fatigue turns sign-off into a rubber stamp.
  • −Validation, PHI contracts and regulatory review cost more than the model.

Check yourself

A router has sensitivity 95% and specificity 90% for urgent messages, which make up 2% of the inbox. Roughly what share of flagged messages are truly urgent?

Check yourself

Which control most directly limits harm from an LLM inventing a normal physical exam in an ambient visit note?

Remember

  • ✓Applied healthcare GenAI is mostly drafting (notes, replies, letters, summaries) for a professional to review; risk rises the closer output gets to a patient decision without a human.
  • ✓Route high-risk inputs away from the model with a classifier tuned for sensitivity; PPV = sens·π / (sens·π + (1 − spec)(1 − π)) shows why most flags are false alarms.
  • ✓In the figure, 99% sensitivity at d′ = 2 flags 64.6% of the inbox; d′ = 3 needs only 28.7%. Classifier quality pays, lowering the threshold does not.
  • ✓Ground drafts in the chart with citations, run deterministic dose and allergy checks, require sign-off, and monitor edit rates and missed cases.
  • ✓Exam results (Flan-PaLM 67.6%, Med-PaLM 2 86.5% on MedQA) are screening evidence, not validation of a workflow.
  • ✓HIPAA/BAA, GDPR Article 9, FDA device rules and the EU AI Act shape deployment; confirm the current rules with counsel.
← back to Generative AI roadmap