Structured outputs & function schemas

Every LLM-powered product hits the same wall eventually. You get the model to give great answers in prose. Users love it. Then engineering says: "Great, now put those answers into a database." Suddenly you need a JSON object with 8 specific fields, and the model returns:

text
Sure! Here's the customer info you requested:
- Name: Alex Chen
- Email: alex@example.com

That's not JSON. It's markdown with a preamble. Your parser explodes. You add "output only JSON" to the prompt. Now it returns:

json
```json
{"name": "Alex Chen", "email": "alex@example.com"}

```

Fenced code block inside markdown fences. Your parser still explodes. You strip the fences with a regex. Ships. Two weeks later a user's name has a curly quote (Alex "The Kid" Chen) and the model returns invalid JSON. Your parser explodes again.

Every one of those bugs disappears the moment you stop trying to convince the model to return valid JSON with words, and start using the API contract that forces it.

There are exactly 3 levels of structured-output extraction. Every production system uses one of them. Once you know which level fits your task, prompt hacking around JSON goes away forever.

## The 3-level extraction ladder — the only diagram you need

```mermaid
flowchart TD
Start([You need structured data from an LLM]) --> Q1
Q1{Is your data schema<br/>fixed and known upfront?}
Q1 -->|NO — free-form text is fine| L0[LEVEL 0<br/>PROMPT-ONLY<br/>Ask nicely. Parse best-effort.<br/>Fine for demos, breaks in prod.]
Q1 -->|YES| Q2{Does the provider support<br/>JSON Schema / Structured Outputs<br/>OpenAI-strict, Anthropic tool-use,<br/>Gemini responseSchema?}
Q2 -->|YES| L2[LEVEL 2<br/>SCHEMA-CONSTRAINED<br/>Provider guarantees valid JSON<br/>matching your schema. Zero retries.]
Q2 -->|NO — older model or self-hosted| L1[LEVEL 1<br/>JSON MODE<br/>Provider guarantees valid JSON<br/>but not schema. You validate + retry.]

Q3{Do you also need the model<br/>to call TOOLS not just return data?}
L2 --> Q3
Q3 -->|YES| Tools[TOOL / FUNCTION CALLING<br/>Same mechanism as Level 2 —<br/>schema constrains BOTH<br/>the tool name AND arguments.]
Q3 -->|NO| Done1([Ship.])
L1 --> Validate[Add Pydantic / Zod validation<br/>+ retry with error message<br/>on schema mismatch.]
Validate --> Done2([Ship.])
L0 --> Warning[[⚠ Do not ship this to prod<br/>without a fallback.]]

classDef l0Node fill:#fee2e2,stroke:#dc2626,color:#7f1d1d
classDef l1Node fill:#fef3c7,stroke:#d97706,color:#78350f
classDef l2Node fill:#dcfce7,stroke:#16a34a,color:#14532d
classDef toolsNode fill:#c4b5fd,stroke:#7c3aed,color:#4c1d95
classDef decisionNode fill:#dbeafe,stroke:#2563eb,color:#1e3a8a
classDef doneNode fill:#e0f2fe,stroke:#0284c7,color:#0c4a6e
class L0 l0Node
class L1,Validate l1Node
class L2 l2Node
class Tools toolsNode
class Q1,Q2,Q3 decisionNode
class Done1,Done2 doneNode
```

Rule: default to Level 2 if your model supports it (GPT-4o, Claude 3.5+, Gemini 1.5+). Fall back to Level 1 with validation + retry for older or self-hosted models. Never rely on Level 0 in production without a downstream fallback (human review, alternative extractor, or graceful degradation).

Now let me walk through each level — the mechanism, the failure modes it prevents, and when it's overkill.

Level 0: Prompt-only — "please return JSON"

What it is: You put "Return your answer as JSON with fields X, Y, Z" in the system prompt. The model tries. Sometimes it succeeds. Sometimes it wraps the JSON in prose. Sometimes it emits invalid JSON (trailing commas, unescaped quotes, hallucinated fields).

Anatomy of a Level 0 prompt:

text
    ═══════════ LEVEL 0 — PROMPT-ONLY ═══════════

    SYSTEM:  You extract structured data from support tickets.
             Return your answer as JSON with these fields:
             - category: string
             - priority: string (low, medium, high)
             - customer_email: string
             Only return the JSON. No preamble.

    USER:    Ticket #12345 from alex@example.com:
             "My subscription renewed but I can't access the pro features."

    MODEL:   {"category": "billing", "priority": "medium",
              "customer_email": "alex@example.com"}

When it works: demos, hackathons, low-stakes internal tools, cases where a bad parse is recoverable.

When it breaks:

1. Preamble drift — the model adds "Sure! Here's the JSON:" before the payload. Your JSON.parse throws.
2. Fenced markdown — the model wraps the JSON in ``json ... ``. Your parser throws unless you strip fences first.
3. Invalid JSON — trailing commas, unescaped quotes in string values, missing brackets when the model runs out of tokens mid-generation.
4. Hallucinated fields — the model adds "confidence": 0.95 or "reasoning": "..." that you didn't ask for.
5. Missing fields — the model omits a required field because the input didn't seem to have it.
6. Type drift — priority is "3" (string) instead of "high" (enum value) or vice versa.

The graveyard is full of production systems that shipped on Level 0 and now have 47 regex patches to handle each failure mode above.

Level 1: JSON mode — provider guarantees valid JSON

What it is: You set a request flag (response_format: {"type": "json_object"} on OpenAI, format: "json" on some clients) that tells the API to guarantee the response parses as JSON. No preamble. No markdown fences. No trailing garbage.

What it does NOT guarantee: that the JSON matches YOUR schema. The model can still return {"answer": "yes"} when you wanted {"category": "...", "priority": "..."}.

Anatomy of a Level 1 request (OpenAI):

python
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Extract fields as JSON..."},
            {"role": "user", "content": "Ticket #12345..."},
        ],
        response_format={"type": "json_object"},   # ← the magic flag
    )
    data = json.loads(response.choices[0].message.content)   # never throws

The API will retry internally (or use constrained decoding — see Chapter 3) to ensure valid JSON syntax. Your parse never explodes on syntax. But it can still explode on shape — e.g. if the model returns a field you weren't expecting, your downstream code breaks.

The right defensive pattern with Level 1:

python
    from pydantic import BaseModel, ValidationError

    class Ticket(BaseModel):
        category: Literal["billing", "technical", "sales", "other"]
        priority: Literal["low", "medium", "high"]
        customer_email: str

    for attempt in range(3):
        response = client.chat.completions.create(..., response_format={"type": "json_object"})
        try:
            ticket = Ticket.model_validate_json(response.choices[0].message.content)
            break   # success
        except ValidationError as e:
            # Feed the error back to the model as a retry
            messages.append({"role": "assistant", "content": response.choices[0].message.content})
            messages.append({"role": "user", "content": f"Validation failed: {e}. Fix and re-emit."})
    else:
        raise RuntimeError("Gave up after 3 retries")

This is the pattern behind LangChain's with_structured_output(method="json_mode") and Instructor's core retry loop.

When it works: you're on a provider that supports JSON mode but not full schema constraints (older GPT-4-turbo, some self-hosted models with structured decoding plugins).

When it breaks: if the model consistently mis-shapes the output, you burn retries. At ~2-3 retries × ~2s latency, this can add 6-10 seconds to your P95.

Level 2: Schema-constrained — provider guarantees valid JSON matching YOUR schema

What it is: You pass a JSON Schema in the request. The provider uses constrained decoding — during generation, at each token step, it only considers tokens that keep the output valid against your schema. Illegal tokens have zero probability. Output is guaranteed to match the schema on the first try.

This is what OpenAI calls "Structured Outputs" (response_format: {"type": "json_schema", "schema": {...}}, since Aug 2024), Anthropic calls "tool use" (with a JSON Schema for tool input), and Google Gemini calls responseSchema.

Anatomy of a Level 2 request (OpenAI Structured Outputs):

python
    from pydantic import BaseModel

    class Ticket(BaseModel):
        category: Literal["billing", "technical", "sales", "other"]
        priority: Literal["low", "medium", "high"]
        customer_email: str

    response = client.beta.chat.completions.parse(
        model="gpt-4o-2024-08-06",
        messages=[
            {"role": "system", "content": "Extract ticket details..."},
            {"role": "user", "content": "Ticket #12345..."},
        ],
        response_format=Ticket,   # ← Pydantic model becomes JSON Schema
    )
    ticket: Ticket = response.choices[0].message.parsed   # already validated

The SDK converts your Pydantic model into a JSON Schema, sends it to the API, and the API's constrained decoder guarantees the output matches. You get back a Pydantic instance directly. No parse. No validate. No retry loop.

Latency cost: ~50-200ms overhead for schema compilation on the first call with a new schema, then negligible on repeat calls (cached).

When it works: modern models on modern providers. Default choice for all new production code in 2025.

When it breaks: deeply nested schemas (>5 levels) can slow constrained decoding measurably; unions with many options can confuse the model semantically even though the syntax is valid; recursive schemas need special handling.

Tool / function calling — Level 2 applied to actions

Function calling is the same mechanism as Level 2, but the JSON output represents a tool invocation rather than a data extraction.

text
    ═══════ FUNCTION CALLING = LEVEL 2 + SEMANTICS ═══════

    You give the model a list of tool schemas:
    ┌──────────────────────────────────────────┐
    │ tools = [                                 │
    │   { name: "get_weather",                  │
    │     parameters: { location: string,       │
    │                    unit: "C" | "F" } }    │
    │   { name: "send_email",                   │
    │     parameters: { to: string,             │
    │                    subject: string,       │
    │                    body: string } }       │
    │ ]                                         │
    └──────────────────────────────────────────┘
                        │
                        ▼
    Model returns EITHER a text response OR a tool call:
    ┌──────────────────────────────────────────┐
    │ { tool_calls: [                           │
    │   { name: "get_weather",                  │
    │     arguments: {location: "SF",           │
    │                 unit: "F"} }              │
    │ ] }                                       │
    └──────────────────────────────────────────┘
    Guaranteed to match ONE of your tool schemas.

Every agent (see A4 journey "Build your first agent") uses function calling as its Action step. Every retrieval-augmented system that decides "search the docs or answer directly?" uses function calling to route.

Which level should you pick? — the 30-second answer

• Free-form text response, downstream is human? Level 0 (prompt-only). No JSON needed at all.
• Structured data, low volume, model is Claude Haiku or older GPT? Level 1 (JSON mode + Pydantic validation + retry).
• Structured data, any volume, modern model (GPT-4o, Claude 3.5+, Gemini 1.5+)? Level 2 (Structured Outputs). Default choice.
• Model needs to call TOOLS, not just return data? Function calling (Level 2 semantic sibling).

The most common mistake that costs teams weeks

Teams prompt-hack at Level 0 for 6 months, then discover Level 2 in a blog post, migrate everything, and delete 400 lines of regex + retry code.

Skip that phase. Start at Level 2 on day one. Fall back to Level 1 only if your model doesn't support it. Fall back to Level 0 only if your task genuinely does not need structured output.

What's next in this journey:

• Chapter 1: JSON Schema deep-dive — the subset of JSON Schema that constrained decoders actually support (no $ref, no patternProperties, no unbounded objects), and how to design schemas that decode fast.
• Chapter 2: Pydantic + Zod patterns — turning your existing domain models into LLM-safe schemas without duplication.
• Chapter 3: How constrained decoding actually works under the hood — token masking, FSMs, GBNF, and why it's more principled than "just retry."
• Chapter 4: Function calling deep-dive — tool schemas, parallel calls, tool result formatting.
• Chapter 5: The failure modes even Level 2 doesn't solve — semantically wrong content, hallucinated enum values, and the eval strategy for structured output quality.

Sources cited in this chapter:

• OpenAI Structured Outputs launch (Aug 2024): platform.openai.com/docs/guides/structured-outputs
• OpenAI JSON Mode: platform.openai.com/docs/guides/text-generation/json-mode
• Anthropic tool use: docs.anthropic.com/claude/docs/tool-use
• Google Gemini responseSchema: ai.google.dev/gemini-api/docs/structured-output
• JSON Schema spec: json-schema.org
• Instructor library (retry-with-validation for Level 1): github.com/jxnl/instructor
• Outlines constrained decoding (open-source Level 2): github.com/outlines-dev/outlines