Reliable Structured Output from LLMs: JSON, Schemas, and Fault Tolerance

As soon as your AI application needs to feed model output into downstream code, which almost all of them do, you cannot avoid making the model emit structured data. And once you have tried, you have almost certainly been burned: ask for JSON and it wraps the JSON in "Sure, here is the result:"; ask for field A and it gives you field B; when it is feeling generous, it may even add a Markdown explanation.

This article covers four tiers in increasing order of reliability. Choose the lowest tier that is sufficient for your use case.

1. Why "Output JSON Only" in the Prompt Is Not Enough

The simplest approach:

Extract the following content as JSON. Output JSON only, with no other text.
{name, age, email}

It works 90% of the time in a demo, but the remaining 10% becomes an incident in production because this approach relies on the model's compliance and enforces nothing. The model may occasionally:

• Add an explanatory sentence.
• Use single quotes or trailing commas, producing invalid JSON.
• Invent a field that is not in the schema.
• Stop halfway through the output.

Prompt-constrained JSON cannot be used on critical paths. It is the lowest tier and is suitable only for prototypes or use cases that can tolerate failures.

2. Tier Two: JSON Mode / Response Format

Mainstream provider APIs now offer a "JSON mode": OpenAI has response_format: {type:"json_object"}, Gemini has responseMimeType, and Anthropic has no native JSON mode but can implement it indirectly through tools.

It guarantees that the output is valid JSON, so it will not be syntactically invalid, but it does not guarantee schema conformance. Field names, types, and nested structures may still deviate from your expectations. In other words, "it is JSON" is guaranteed; "it is the JSON you wanted" is not.

Suitable for use cases with simple schemas where field variations are acceptable. It is one tier stronger than a prompt alone, but do not expect it to guarantee the structure.

3. Tier Three: Function Calling / Tool Use (Recommended Default)

This is currently the most practical way to obtain structured output in production. You define a "tool" whose parameter schema is the structure you want:

const tools = [{
  type: "function",
  function: {
    name: "extract_person",
    description: "Extract a person's info from the text.",
    parameters: {
      type: "object",
      required: ["name", "email"],
      properties: {
        name:  { type: "string" },
        age:   { type: "number" },
        email: { type: "string", format: "email" },
        role:  { type: "string", enum: ["user", "admin"] },
      },
    },
  },
}];

The model is trained to "fill in arguments when calling a tool," and those arguments are inherently structured. Providers apply constraints along this path, so field names, types, and enums generally line up. The tool_call.arguments you receive is JSON that conforms to the schema.

Why is it stronger than JSON mode?

• The schema provides explicit constraints: required, enum, and type all take effect.
• The model naturally understands "filling out a form" better than "emitting JSON."
• Providers have specifically reinforced this path during training.

Suitable for the vast majority of production use cases. Even if you do not actually need to "call a tool," you can use function calling to obtain structured output. This is a standard industry technique.

4. Tier Four: Constrained Decoding (Grammar / Outlines)

If you need close to a 100% schema guarantee, such as mandatory enum values, numeric ranges, or nested structures, use constrained decoding. A grammar enforces the structure during decoding, so the model can only choose valid tokens.

Representative projects include Outlines, lm-format-enforcer, and grammar constraints such as GBNF in major inference frameworks. Their principle is straightforward: you provide a schema or grammar, and at each decoding step the decoder masks tokens that would make the output invalid, guaranteeing structural conformance by construction.

# Outlines example (local model)
from outlines import models, generate
model = models.transformers("mistral-7b")
schema = '{"name": str, "age": int, "email": str}'
generator = generate.json(model, schema)
result = generator("Extract: ...")   # 100% schema-compliant

The tradeoff is that it is only practical for self-hosted/open-source models, because closed-source APIs do not expose the decoding layer. This tier is therefore suitable when you run your own model and require extremely high structural correctness.

5. Parsing and Fault Tolerance: Every Tier Needs Them

Whichever tier you choose, downstream parsing still needs fault tolerance because unexpected output will always occur in practice:

function safeParse(raw: string): Record<string, unknown> {
  // 1. Parse directly
  try { return JSON.parse(raw); } catch {}

  // 2. Extract the first {...} or [...] fragment (strip explanatory text)
  const m = raw.match(/[{[][\s\S]*[}\]]/);
  if (m) { try { return JSON.parse(m[0]); } catch {} }

  // 3. Fix common issues (trailing commas and single quotes), then retry
  try { return JSON.parse(m[0].replace(/,\s*([}\]])/g, "$1").replace(/'/g, '"')); } catch {}

  throw new Error("unparseable model output");
}

Add validation with zod or ajv. Successfully parsing the output does not mean it conforms to the schema, so validate it with a schema validator. If validation fails, retry with the error information, feeding a message such as "your output is missing the email field" back to the model so it can correct the result.

This fallback chain of "parse → extract → repair → validate → retry with the error" is essential for pushing any tier toward production reliability. Do not assume that the model will always produce correct output. Assume that it will occasionally fail, and make sure the code can recover.

6. Choose a Tier by Use Case

Summary

For structured LLM output, reliability increases tier by tier from "asking nicely" to "enforcement at the decoding layer":

• Prompt constraints: weakest; use for prototypes.
• JSON mode: guarantees syntax, not structure.
• Function calling: the production default; the schema generally lines up.
• Constrained decoding: close to a 100% guarantee, but requires a self-hosted model.

Whatever tier you choose, downstream code needs a fault-tolerance chain of parse → repair → validate → retry with the error. In production systems, "the model occasionally produces incorrect output" is inevitable. Engineering is responsible for making the system recover even when that happens. This is what separates structured output in a demo from structured output in production.