Definition
Structured output means making a language model return data in a machine-readable format, usually a JSON object that matches a given JSON Schema, instead of free prose. There are three levels. Prompt-only asks for JSON in the instructions. JSON mode (OpenAI, November 2023, response_format: {"type": "json_object"}) guarantees the output parses. Schema-constrained output (OpenAI Structured Outputs, August 2024, "strict": true; open-source libraries such as Outlines, after Willard and Louf, 2023) also guarantees the output matches the schema.
Most LLM features end in code, not in a chat window. An extractor fills a database row, a classifier routes a ticket, an agent calls a function. That code needs fields with known names and types. A model that wraps its JSON in markdown fences, adds "Sure! Here is the JSON:", drops a required key or invents an enum value breaks the pipeline, and at scale even a 2% failure rate is thousands of failed requests a day.
Constrained output fixes the form at decode time. Each step masks every token that would make the output leave the schema's grammar, then renormalises over the rest. The token-mask mechanism (finite-state machines, grammars, mask caching) is covered in Constrained decoding. This article covers the applied layer: which mode guarantees what, how to write schemas that work, and how to validate and retry.
What each mode guarantees
API behaviour (OpenAI, as introduced in 2023–2024; other providers offer similar options)
- JSON mode
{"type": "json_object"}guarantees syntactically valid JSON, with no schema. The messages must mention "JSON", or the API returns an error. Without that instruction the model may emit whitespace until it hits the token limit.- Structured Outputs
{"type": "json_schema", "json_schema": {"name": …, "strict": true, "schema": …}}. At launch OpenAI reported that gpt-4o-2024-08-06 scored 100% on its complex-schema benchmark, against under 40% for gpt-4-0613 with prompting alone.- Strict-mode schema rules
- Only a subset of JSON Schema is accepted. Every property must be listed in
required, and every object needs"additionalProperties": false. An optional field becomes a union withnull. Keys are generated in schema order. - Refusals and truncation
- A safety refusal comes back in a separate
refusalfield, not as schema-shaped JSON. Output cut off at the token limit (finish_reason: "length") is invalid JSON whatever the mode. - Open-source stacks
- Outlines compiles regexes and JSON Schemas into finite-state machines over the vocabulary. llama.cpp uses GBNF grammars. vLLM and SGLang expose guided decoding, with backends such as XGrammar (Dong et al., 2024).
- Function calling
- A tool's
parametersis a JSON Schema, and the same strict flag applies (see LLM APIs and function calling).
Even strict mode guarantees only the form. A schema-valid {"category": "bug"} for a billing email is still wrong. Business rules that the schema does not encode, or that the provider's subset does not support (ranges, cross-field checks, existence of an ID in your database), still need a validator in code.
Validate and retry
Model each attempt as passing parse (probability ) and each of schema fields independently (probability each). Then:
- chance the text does not parse (fences, preamble, truncation); JSON and strict modes set except for truncation
- chance one field breaks the schema (missing, wrong type, value outside the enum); strict mode sets
- chance of a valid object within attempts when each retry is independent
The product is the reason prompt-only JSON degrades on big schemas: at , 8 fields give 78.4% and 20 fields give 54.4%. Retrying helps quickly but costs calls and latency. Retries with the error message fed back usually do better than independent ones, so treat as a floor.
| Quantity | Computation | Value |
|---|---|---|
| All fields pass | 0.97⁸ | 0.7837 |
| First attempt valid p₁ | 0.94 × 0.7837 | 0.7367 |
| All three attempts fail | (1 − 0.7367)³ = 0.2633³ | 0.0182 |
| Valid within 3 tries P₃ | 1 − 0.0182 | 0.9818 |
| Expected calls | 0.9818 / 0.7367 | 1.333 |
| Valid and correct | 0.9818 × (1 − 0.04) | 0.9425 |
1from typing import Literal2from openai import OpenAI3from pydantic import BaseModel, ConfigDict, ValidationError, field_validator45class Ticket(BaseModel):6 model_config = ConfigDict(extra="forbid") # -> "additionalProperties": false7 reasoning: str # generated first: think, then answer8 category: Literal["billing", "bug", "other"]9 priority: int10 summary: str1112 @field_validator("priority")13 @classmethod14 def in_range(cls, v: int) -> int: # business rule outside the schema15 if not 1 <= v <= 5:16 raise ValueError("priority must be between 1 and 5")17 return v1819client = OpenAI()20FORMAT = {"type": "json_schema",21 "json_schema": {"name": "ticket", "strict": True,22 "schema": Ticket.model_json_schema()}}2324def extract(email: str, k: int = 3) -> Ticket:25 messages = [{"role": "system", "content": "Extract a support ticket from the email."},26 {"role": "user", "content": email}]27 for _ in range(k):28 msg = client.chat.completions.create(29 model="gpt-4o-mini", messages=messages, response_format=FORMAT,30 ).choices[0].message31 if msg.refusal: # refusals are not schema-shaped32 raise RuntimeError(msg.refusal)33 try:34 return Ticket.model_validate_json(msg.content)35 except ValidationError as err: # feed the error back, try again36 messages += [{"role": "assistant", "content": msg.content},37 {"role": "user", "content": f"Invalid: {err}. Return corrected JSON."}]38 raise RuntimeError(f"no valid ticket after {k} attempts")Schema design that works with left-to-right decoding
- 1Order fields so dependencies come first. The model writes keys in schema order and cannot revise them. Put a short
reasoningorevidencefield beforeanswer, or the answer is committed before any reasoning exists. - 2Prefer enums to free strings for anything code branches on. A closed set of values is easy to mask and easy to evaluate.
- 3Make absence explicit. Use
"type": ["string", "null"]and tell the model to return null when the source lacks the fact. Otherwise a required field invites a made-up value. - 4Describe fields.
descriptionstrings in the schema reach the model as instructions and cost far less than a failed parse. - 5Keep it flat and small. Deep nesting and long schemas raise for prompt-only output and the compile time for constrained output. Many APIs compile a new schema on first use, so the first request is slower.
Use when
- +Strict schemas drive parse and schema failures to zero within the supported subset: no repair heuristics needed.
- +Fewer retries means lower cost and tail latency than prompt-only JSON.
- +One schema serves as the API contract, the model instruction and the validator.
Avoid when
- −Guarantees form, not truth: wrong values, hallucinated fields filled with plausible text.
- −Only a subset of JSON Schema is supported; ranges and cross-field rules often still need code.
- −Masking can hurt reasoning quality, and new schemas add compile latency on first use.
- −Refusals and token-limit truncation still produce non-schema output that code must handle.
Check yourself
You enable JSON mode (json_object) for a ticket extractor. Which failure can still happen?
Check yourself
Prompt-only JSON, per-field error q = 3%, no syntax errors. Roughly what share of 20-field objects pass on the first try?
Remember
- ✓Prompt-only < JSON mode (parses) < strict schema (parses and fits the schema). None of them guarantees correct values.
- ✓Validity compounds: , and independent retries give at expected calls.
- ✓Strict mode needs every key required, additionalProperties false, nullable unions for optional fields, and a supported keyword subset.
- ✓Order fields for left-to-right generation (reasoning before answer), prefer enums, make absence explicit with null.
- ✓Always validate in code (business rules, refusals, truncation) and retry with the error message; see Constrained decoding for the mask itself.