Definition
Generative AI in healthcare is the use of generative models, mostly large language models and speech-to-text, to draft clinical and administrative text for a clinician or staff member to review. The work covers visit notes, patient-message replies, discharge summaries, referral and prior-authorisation letters, and answers to questions about guidelines. The field's reference results are Med-PaLM (Singhal et al., "Large language models encode clinical knowledge", Nature 2023) and Med-PaLM 2 (Singhal et al., 2023). Both scored USMLE-style exam questions; neither measured patient outcomes.
The problem is clinician time. Documentation and inbox work take a large share of a physician's day, and that work is mostly text: summarising a visit, answering refill and results questions, writing letters. Generative models are good at drafting such text. They are not reliable enough to make clinical decisions alone, so the design question is always the same: who reads the output before it affects a patient?
In production systems the model sits inside the electronic health record (EHR) workflow, not in a chat window. It gets its facts from the patient's chart by retrieval (see RAG architecture). It writes into a draft field, and a clinician edits and signs. A separate classifier decides which inputs the model should not touch at all.
Where it is used
Where it's used
Ambient documentation
Speech recognition (see Whisper) plus Speaker diarization turn the visit audio into a transcript; an LLM drafts the note sections. The clinician reviews and signs. Patient consent and audio-retention rules apply.
Patient-message drafts
Replies to portal messages about refills, results and logistics, grounded in the chart and sent only after clinician edit and sign-off.
Summaries and letters
Discharge summaries, referral letters, prior-authorisation letters and plain-language versions of notes (see Document summarization).
Guideline and coding questions
Retrieval over approved guidelines and formularies, with citations, for clinicians and coders. Answers are references to read, not orders.
Risk depends on who acts on the output
| Use | Who reads it first | Typical failure | Minimum controls |
|---|---|---|---|
| Admin letter, scheduling text | Staff | Wrong date or code | Spot-check sample, templates |
| Visit note draft | Signing clinician | Invented exam finding, omitted symptom | Review before signing, source transcript kept for checking |
| Patient reply draft | Clinician, then patient | Unsafe advice, missed urgency | Urgency routing, grounding, sign-off, AI-assisted label |
| Diagnosis or treatment suggestion | Clinician acting on it | Plausible but wrong recommendation | May be a regulated medical device; prospective validation |
| Patient-facing advice, no human | Patient | Harm with no one in the loop | Generally out of scope without regulatory clearance |
Routing: sensitivity first, then workload
The most important control is the router. It is a classifier, often a small fine-tuned model or an LLM prompted for a score. A missed urgent message (a false negative) can harm a patient, while a false alarm costs nurse time. So the threshold is set for a target sensitivity on labelled validation data, and the cost shows up as workload. Base rates make this costly. Urgent messages are rare, so most flags are false alarms (see Precision, recall & ROC):
- prevalence: the share of messages that are truly urgent
- positive predictive value: the chance that a flagged message is really urgent
- urgent caught, urgent missed, routine flagged, routine passed to drafting
The figure uses the binormal model: routine scores follow and urgent scores follow . Then , , and . The threshold for 99% sensitivity is .
| Setting | t | Sens. | Spec. | Missed (FN) | False alarms (FP) | PPV | Flagged |
|---|---|---|---|---|---|---|---|
| d′ = 2, default | 1.00 | 84.1% | 84.1% | 7.9 | 150.7 | 21.8% | 19.3% |
| d′ = 2, 99% sensitivity | −0.33 | 99.0% | 37.2% | 0.5 | 596.5 | 7.7% | 64.6% |
| d′ = 3, 99% sensitivity | 0.67 | 99.0% | 75.0% | 0.5 | 237.8 | 17.2% | 28.7% |
Reaching 99% sensitivity with the weaker classifier sends almost two thirds of the inbox to nurses. That removes most of the time saved. A better classifier or a narrower scope (say, only refill requests) is what makes the system pay off. A lower threshold does not.
1import numpy as np23def threshold_for_sensitivity(scores, labels, target=0.99):4 """Highest threshold t whose validation sensitivity is at least `target`."""5 pos = np.sort(scores[labels == 1])6 k = int(np.floor((1 - target) * len(pos))) # urgent messages we may miss7 return pos[k] # flag when score >= t89rng = np.random.default_rng(0)10n_urgent, n_routine = 400, 7600 # 5% urgent, like the figure11scores = np.r_[rng.normal(2.0, 1, n_urgent), rng.normal(0.0, 1, n_routine)]12labels = np.r_[np.ones(n_urgent, int), np.zeros(n_routine, int)]1314t = threshold_for_sensitivity(scores, labels)15flag = scores >= t16print(f"t = {t:.2f}") # t = -0.3717print(f"sensitivity {flag[labels == 1].mean():.3f}") # 0.990 (4 of 400 missed)18print(f"flagged {flag.mean():.1%}, PPV {labels[flag].mean():.3f}") # 66.0%, 0.075What the evidence shows
Published results (each in its own setting)
- Med-PaLM (2023)
- Flan-PaLM 540B with prompting scored 67.6% on MedQA (USMLE-style, 4 options), the first model past the roughly 60% pass mark. Clinician raters still found gaps, which led to instruction prompt tuning in Med-PaLM.
- Med-PaLM 2 (2023)
- 86.5% on MedQA. In pairwise ratings on consumer medical questions, physicians preferred its long-form answers to physicians' answers on most quality axes.
- GPT-4 (Nori et al., 2023)
- Exceeded the USMLE passing score by over 20 points with no medical fine-tuning or special prompting.
- Ayers et al., JAMA Intern. Med. 2023
- 195 public r/AskDocs questions: evaluators preferred ChatGPT's replies to physicians' in 78.6% of 585 evaluations, and rated them higher for quality and empathy. The physician replies were short forum posts, not clinic messages.
- What they do not show
- Multiple-choice vignettes give the model a clean summary of the case. Real use requires finding facts in a messy chart, knowing what is missing, and outcome evidence from prospective studies.
Treat benchmark scores as a screening test for a model, not as validation of a use case. Validation means your own data and your own task. Clinicians rate the drafts for errors and omissions (see Human evaluation and arena-style ranking), claims are checked against the source chart (see Hallucination and groundedness evaluation), and edit rates and incident reports are watched after launch.
Failure modes specific to clinical text
Controls, in the order a request meets them
- 1PHI boundary. In the US, HIPAA covers protected health information; a vendor processing it for a covered entity needs a business associate agreement (BAA). In the EU, health data is a special category under GDPR Article 9. Send the minimum necessary fields, and check whether the vendor retains prompts or uses them for training (see Privacy and data leakage).
- 2Scope and routing. Decide which inputs never get a draft (chest pain, suicidal ideation, new neurological symptoms) and route them by a classifier tuned for sensitivity, plus keyword rules as a backstop.
- 3Grounding. Retrieve the medication list, allergies, problems and recent results. Instruct the model to cite chart rows, and reject drafts with uncited clinical claims. Hallucinated findings, such as a normal exam that was never done, are the most cited failure of ambient notes (see Hallucination).
- 4Hard checks. Run deterministic checks for doses against a formulary, drug–allergy conflicts, and no new diagnosis or dose change in a patient reply. Use Structured output (JSON mode) to make the fields checkable.
- 5Human sign-off. The clinician sees the draft and its sources and edits, signs or discards it. Label the reply as AI-assisted where policy or law requires.
- 6Monitoring. Track edit distance between draft and sent text, the discard rate, sampled audits of routed and unrouted messages, and differences across patient groups (see Bias and fairness in generative models).
Use when
- +Drafting cuts time on documentation and inbox work while the clinician keeps the decision.
- +Chart grounding plus citations makes each claim checkable in seconds.
- +Plain-language rewrites and translations make notes easier for patients to read.
Avoid when
- −Errors are fluent and plausible; an omitted symptom is harder to spot than a wrong one.
- −High-sensitivity routing can flag most of the inbox and wipe out the time saved.
- −Review fatigue turns sign-off into a rubber stamp.
- −Validation, PHI contracts and regulatory review cost more than the model.
Check yourself
A router has sensitivity 95% and specificity 90% for urgent messages, which make up 2% of the inbox. Roughly what share of flagged messages are truly urgent?
Check yourself
Which control most directly limits harm from an LLM inventing a normal physical exam in an ambient visit note?
Remember
- ✓Applied healthcare GenAI is mostly drafting (notes, replies, letters, summaries) for a professional to review; risk rises the closer output gets to a patient decision without a human.
- ✓Route high-risk inputs away from the model with a classifier tuned for sensitivity; PPV = sens·π / (sens·π + (1 − spec)(1 − π)) shows why most flags are false alarms.
- ✓In the figure, 99% sensitivity at d′ = 2 flags 64.6% of the inbox; d′ = 3 needs only 28.7%. Classifier quality pays, lowering the threshold does not.
- ✓Ground drafts in the chart with citations, run deterministic dose and allergy checks, require sign-off, and monitor edit rates and missed cases.
- ✓Exam results (Flan-PaLM 67.6%, Med-PaLM 2 86.5% on MedQA) are screening evidence, not validation of a workflow.
- ✓HIPAA/BAA, GDPR Article 9, FDA device rules and the EU AI Act shape deployment; confirm the current rules with counsel.