# Structured output (JSON mode)

> JSON mode guarantees parseable JSON and strict schema mode also guarantees the schema by masking tokens, but neither makes values right, so validate and retry: success within k tries is 1 − (1 − p)^k.

Source: https://www.mlfoundry.in/learn/structured-output  
Section: Generative AI › Prompt Engineering  
Level: core · 7 min read · updated 2026-10-21

**Structured output:** means making a language model return data in a machine-readable format, usually a JSON object that matches a given JSON Schema, instead of free prose. There are three levels. **Prompt-only** asks for JSON in the instructions. **JSON mode** (OpenAI, November 2023, `response_format: {"type": "json_object"}`) guarantees the output parses. **Schema-constrained output** (OpenAI Structured Outputs, August 2024, `"strict": true`; open-source libraries such as Outlines, after Willard and Louf, 2023) also guarantees the output matches the schema.

Most LLM features end in code, not in a chat window. An extractor fills a database row, a classifier routes a ticket, an agent calls a function. That code needs fields with known names and types. A model that wraps its JSON in markdown fences, adds "Sure! Here is the JSON:", drops a required key or invents an enum value breaks the pipeline, and at scale even a 2% failure rate is thousands of failed requests a day.

Constrained output fixes the form at decode time. Each step masks every token that would make the output leave the schema's grammar, then renormalises over the rest. The token-mask mechanism (finite-state machines, grammars, mask caching) is covered in **Constrained decoding**. This article covers the applied layer: which mode guarantees what, how to write schemas that work, and how to validate and retry.

> Figure: A Pydantic schema for a support ticket becomes a json_schema response format with strict: true. While decoding, after the prefix {"category": " only billing, bug and other survive the mask. The 0.88 of probability mass they hold is renormalised to 0.693 / 0.250 / 0.057, and urgent and Sure go to 0. The output parses and fits the schema, but the application's validator still rejects priority 6, a rule the schema did not carry, and triggers a retry with the error text.

## What each mode guarantees

**API behaviour (OpenAI, as introduced in 2023–2024; other providers offer similar options)**

- **JSON mode:** `{"type": "json_object"}` guarantees syntactically valid JSON, with no schema. The messages must mention "JSON", or the API returns an error. Without that instruction the model may emit whitespace until it hits the token limit.
- **Structured Outputs:** `{"type": "json_schema", "json_schema": {"name": …, "strict": true, "schema": …}}`. At launch OpenAI reported that gpt-4o-2024-08-06 scored 100% on its complex-schema benchmark, against under 40% for gpt-4-0613 with prompting alone.
- **Strict-mode schema rules:** Only a subset of JSON Schema is accepted. Every property must be listed in `required`, and every object needs `"additionalProperties": false`. An optional field becomes a union with `null`. Keys are generated in schema order.
- **Refusals and truncation:** A safety refusal comes back in a separate `refusal` field, not as schema-shaped JSON. Output cut off at the token limit (`finish_reason: "length"`) is invalid JSON whatever the mode.
- **Open-source stacks:** Outlines compiles regexes and JSON Schemas into finite-state machines over the vocabulary. llama.cpp uses GBNF grammars. vLLM and SGLang expose guided decoding, with backends such as XGrammar (Dong et al., 2024).
- **Function calling:** A tool's `parameters` is a JSON Schema, and the same strict flag applies (see **LLM APIs and function calling**).

Even strict mode guarantees only the **form**. A schema-valid `{"category": "bug"}` for a billing email is still wrong. Business rules that the schema does not encode, or that the provider's subset does not support (ranges, cross-field checks, existence of an ID in your database), still need a validator in code.

## Validate and retry

Model each attempt as passing parse (probability $1-s$) and each of $n$ schema fields independently (probability $1-q$ each). Then:

$$
p_1=(1-s)(1-q)^{n},\qquad P_k=1-(1-p_1)^{k},\qquad \mathbb E[\text{calls}]=\sum_{j=0}^{k-1}(1-p_1)^j=\frac{P_k}{p_1}
$$

- $s$: chance the text does not parse (fences, preamble, truncation); JSON and strict modes set $s=0$ except for truncation
- $q$: chance one field breaks the schema (missing, wrong type, value outside the enum); strict mode sets $q=0$
- $P_k$: chance of a valid object within $k$ attempts when each retry is independent

The product $(1-q)^n$ is the reason prompt-only JSON degrades on big schemas: at $q=3\%$, 8 fields give 78.4% and 20 fields give 54.4%. Retrying helps quickly but costs calls and latency. Retries with the error message fed back usually do better than independent ones, so treat $P_k$ as a floor.

> Figure: 1,000 simulated requests per mode, defaults n = 8, q = 3%, s = 6%, k = 3, e = 4%. Prompt-only: p₁ = 73.7% and P(≤3) = 98.2% in theory; simulated, 74.6% pass on the first try, 24.1% after a retry, 1.3% fail all three, at 1.323 calls per request. JSON mode: 78.1% / 21.1% / 0.8% at 1.259 calls. Strict schema: 100% on one call, yet 3.7% of objects still carry a wrong value. Set n = 20: p₁ falls to 51.1% and 54.4%, and with k = 1 about half the requests fail (48.5% and 46.8% simulated).

Worked example, prompt-only, n = 8 fields, q = 3%, s = 6%, k = 3, wrong-value rate e = 4%

| Quantity | Computation | Value |
| --- | --- | --- |
| All fields pass | 0.97⁸ | 0.7837 |
| First attempt valid p₁ | 0.94 × 0.7837 | 0.7367 |
| All three attempts fail | (1 − 0.7367)³ = 0.2633³ | 0.0182 |
| Valid within 3 tries P₃ | 1 − 0.0182 | 0.9818 |
| Expected calls | 0.9818 / 0.7367 | 1.333 |
| Valid and correct | 0.9818 × (1 − 0.04) | 0.9425 |

```python
from typing import Literal
from openai import OpenAI
from pydantic import BaseModel, ConfigDict, ValidationError, field_validator

class Ticket(BaseModel):
    model_config = ConfigDict(extra="forbid")       # -> "additionalProperties": false
    reasoning: str                                  # generated first: think, then answer
    category: Literal["billing", "bug", "other"]
    priority: int
    summary: str

    @field_validator("priority")
    @classmethod
    def in_range(cls, v: int) -> int:               # business rule outside the schema
        if not 1 <= v <= 5:
            raise ValueError("priority must be between 1 and 5")
        return v

client = OpenAI()
FORMAT = {"type": "json_schema",
          "json_schema": {"name": "ticket", "strict": True,
                          "schema": Ticket.model_json_schema()}}

def extract(email: str, k: int = 3) -> Ticket:
    messages = [{"role": "system", "content": "Extract a support ticket from the email."},
                {"role": "user", "content": email}]
    for _ in range(k):
        msg = client.chat.completions.create(
            model="gpt-4o-mini", messages=messages, response_format=FORMAT,
        ).choices[0].message
        if msg.refusal:                             # refusals are not schema-shaped
            raise RuntimeError(msg.refusal)
        try:
            return Ticket.model_validate_json(msg.content)
        except ValidationError as err:              # feed the error back, try again
            messages += [{"role": "assistant", "content": msg.content},
                         {"role": "user", "content": f"Invalid: {err}. Return corrected JSON."}]
    raise RuntimeError(f"no valid ticket after {k} attempts")
```

Strict schema plus a validator and error-feedback retry (not run here: needs openai, pydantic v2 and an API key; the model name is illustrative). A Pydantic schema may need small changes to fit a provider's strict subset; SDK helpers such as OpenAI's parse method make them for you.

## Schema design that works with left-to-right decoding

1. **Order fields so dependencies come first.** The model writes keys in schema order and cannot revise them. Put a short `reasoning` or `evidence` field before `answer`, or the answer is committed before any reasoning exists.
2. **Prefer enums to free strings** for anything code branches on. A closed set of values is easy to mask and easy to evaluate.
3. **Make absence explicit.** Use `"type": ["string", "null"]` and tell the model to return null when the source lacks the fact. Otherwise a required field invites a made-up value.
4. **Describe fields.** `description` strings in the schema reach the model as instructions and cost far less than a failed parse.
5. **Keep it flat and small.** Deep nesting and long schemas raise $q$ for prompt-only output and the compile time for constrained output. Many APIs compile a new schema on first use, so the first request is slower.

> **Format constraints can cost accuracy:** Tam et al. (2024), "Let Me Speak Freely?", found that strict format constraints can lower LLM performance on reasoning tasks compared with free-form answers. A mask also distorts the distribution: the model's preferred token may be blocked, and the renormalised mass can steer it down an unlikely path. Mitigations: give it a reasoning field first, or answer in free text and convert to JSON in a second, cheap call. Measure accuracy, not just validity.

**Strengths**

- Strict schemas drive parse and schema failures to zero within the supported subset: no repair heuristics needed.
- Fewer retries means lower cost and tail latency than prompt-only JSON.
- One schema serves as the API contract, the model instruction and the validator.

**Limitations**

- Guarantees form, not truth: wrong values, hallucinated fields filled with plausible text.
- Only a subset of JSON Schema is supported; ranges and cross-field rules often still need code.
- Masking can hurt reasoning quality, and new schemas add compile latency on first use.
- Refusals and token-limit truncation still produce non-schema output that code must handle.

**Question:** You enable JSON mode (json_object) for a ticket extractor. Which failure can still happen?

- A. The reply starts with "Here is the JSON:"
- B. The reply is missing the required "priority" key
- C. The reply has an unclosed brace on a normal, untruncated response
- D. None: JSON mode enforces the schema

**Answer:** B. The reply is missing the required "priority" key — JSON mode guarantees only that the output parses as JSON. It knows nothing about your keys or types. A missing field is a schema violation, which needs strict schema mode or validation and retry.

**Question:** Prompt-only JSON, per-field error q = 3%, no syntax errors. Roughly what share of 20-field objects pass on the first try?

- A. 97%
- B. 78%
- C. 54%
- D. 40%

**Answer:** C. 54% — $0.97^{20}=0.544$. Field errors compound, so a schema that is fine with 3 fields fails almost half the time with 20, which is why big schemas benefit most from constrained decoding.

**Key points**

- Prompt-only < JSON mode (parses) < strict schema (parses and fits the schema). None of them guarantees correct values.
- Validity compounds: $p_1=(1-s)(1-q)^n$, and $k$ independent retries give $1-(1-p_1)^k$ at $P_k/p_1$ expected calls.
- Strict mode needs every key required, additionalProperties false, nullable unions for optional fields, and a supported keyword subset.
- Order fields for left-to-right generation (reasoning before answer), prefer enums, make absence explicit with null.
- Always validate in code (business rules, refusals, truncation) and retry with the error message; see **Constrained decoding** for the mask itself.

---

From MLFoundry (https://www.mlfoundry.in). The web page adds interactive figures and diagrams.
