Structured LLM Outputs: JSON Schemas and Function Calling

17 Nov 2026

9K

35K

Structured LLM Outputs: JSON Schemas and Function Calling

Large language models are conversational by design, which makes them unreliable as data sources. Ask a model to extract contact details from an email, summarize a support ticket into fields, or classify a product inquiry, and you get prose: courteous, variable, and full of markdown fences. Applications that consume model output need contracts, not chatter. Structured outputs close that gap by constraining model generation to a JSON schema you define, turning the model from a chat partner into a dependable data producer.

This article explains how structured outputs work, when to use function calling versus a structured response format, how to design schemas models actually follow, and how to handle failures gracefully.

Why Free-Form Output Breaks Pipelines

Free-form generation optimizes for fluency, not consistency. The same prompt can return "John Doe" one run and "Doe, John" the next, wrap dates in "as of March 3" on some runs and ISO strings on others, and sprinkle conversational filler around the payload. Downstream code that parses regexes or splits on markers breaks silently, and teams respond with retry loops that multiply latency and cost without fixing the root problem.

A schema acts as a data contract between your prompt and your application. When the model must honor it, every run returns the same keys, types, and shapes, so the rest of your pipeline can consume the result directly.

Structured Outputs versus JSON Mode

JSON mode tells the model to emit valid JSON, but it says nothing about what that JSON should contain. A model in JSON mode can still omit required fields, invent keys, or return strings where numbers are expected.

Structured outputs go further. With a JSON Schema and strict mode enabled, the model must produce exactly the keys you declared, with the types you declared, and nothing extra. additionalProperties: false blocks surprise fields; required guarantees presence; enums restrict values to a fixed list. The result is valid JSON that also conforms to your contract.

Structured outputs are available from GPT-4o onward. JSON mode remains useful for older models, but for schema-bound work, structured outputs are the stricter, safer default.

Two Ways to Request Structure

You can request structured output in two places, and the choice depends on intent:

  1. Function calling (tools) with strict: true — best when the model's output should trigger something in your application: updating a record, querying a database, or composing a UI card. The model selects a function and fills its arguments according to your schema.
  2. response_format with type json_schema — best when the structure belongs to the model's user-facing answer itself: a quiz generator returning questions and answers, or a tutor returning an explanation plus practice problems.

Use function calling when structure feeds an action; use response_format when structure feeds the reader.

Designing Schemas Models Follow

A model-friendly schema is flat, explicit, and restrictive:

  • Name keys plainly (customer_email, not e). Titles and descriptions help the model understand intent.
  • Prefer one level of nesting. Deeply nested objects increase the chance of skipped or malformed fields.
  • Use supported JSON Schema types: string, number, boolean, integer, object, array, enum, and anyOf where supported.
  • Mark every field required and set additionalProperties: false. Strict mode treats any missing or extra field as a violation.
  • Use enum for fixed categories (priority: low, medium, high) and format or pattern for strings such as email, date-time, or uuid.
  • Avoid unsupported features such as dependencies, if/then, or unevaluatedProperties.

Example schema for extracting a support contact:

{
  "name": "extract_support_contact",
  "strict": true,
  "parameters": {
    "type": "object",
    "properties": {
      "name": {
        "type": "string",
        "description": "Full name of the person contacting support"
      },
      "email": {
        "type": "string",
        "format": "email"
      },
      "issue": {
        "type": "string",
        "enum": ["billing", "technical", "account", "other"]
      },
      "urgency": {
        "type": "integer",
        "minimum": 1,
        "maximum": 5
      }
    },
    "required": ["name", "email", "issue", "urgency"],
    "additionalProperties": false
  }
}

Code Example: Extracting Structured Data

Here is a minimal Python example using the OpenAI client with strict function calling:

from openai import OpenAI
import json

client = OpenAI()

schema = {
  "name": "extract_support_contact",
  "strict": True,
  "parameters": {
    "type": "object",
    "properties": {
      "name": {"type": "string"},
      "email": {"type": "string", "format": "email"},
      "issue": {
        "type": "string",
        "enum": ["billing", "technical", "account", "other"]
      },
      "urgency": {
        "type": "integer",
        "minimum": 1,
        "maximum": 5
      }
    },
    "required": ["name", "email", "issue", "urgency"],
    "additionalProperties": False
  }
}

response = client.chat.completions.create(
  model="gpt-4o",
  messages=[{
    "role": "user",
    "content": "Hi, I'm Maria, [email protected], and my billing card keeps declining."
  }],
  tools=[{"type": "function", "function": schema}],
  tool_choice={
    "type": "function",
    "function": {"name": "extract_support_contact"}
  },
  parallel_tool_calls=False,
)

arguments = response.choices[0].message.tool_calls[0].function.arguments
data = json.loads(arguments)

Note parallel_tool_calls=False: structured outputs and parallel tool calls do not combine, so disable it. The arguments string is valid JSON by contract, but wrap json.loads in error handling anyway—network issues and refusals still happen.

Refusals, Errors, and Validation

A schema guarantees shape, not truth. A model can return a syntactically perfect object with a hallucinated email or an urgency of 5 for a trivial request. Validate business rules separately: date ranges, referential integrity, whether an email actually belongs to the customer.

Structured outputs do make failures programmatic. Refusals surface as structured refusal messages rather than garbled JSON, and missing tool calls are easier to detect than a broken regex match. Log refusals, surface them to users when appropriate, and never retry a refused request blindly.

Common Mistakes

  • Nesting objects four levels deep. Keep schemas shallow.
  • Leaving fields optional. In strict mode, omission fails validation; mark what you need.
  • Using loose anyOf everywhere. It weakens the contract the model must honor.
  • Running structured outputs with parallel tool calls enabled. Disable parallel_tool_calls.
  • Confusing JSON mode with structured outputs. JSON mode validates syntax, not shape.

Trade-offs

Structured outputs cost a little more design work up front and reduce the model's freedom to improvise—which is the point for data tasks and the wrong fit for creative writing, brainstorming, or open-ended explanation. Schema changes also need versioning: adding a required field breaks consumers expecting the old shape. Treat the schema as part of your API and evolve it deliberately.

Use evaluations, not vibes, to decide. Run a sample of real inputs through your schema before shipping, and measure refusal rates and field-completion rates.

Production Checklist

  • Pick one recurring extraction or classification task first.
  • Write a strict schema with required fields and additionalProperties: false.
  • Test with real inputs, including edge cases and refusals.
  • Disable parallel tool calls when using structured functions.
  • Add business-rule validation after parsing.
  • Log schema violations and refusals for continuous improvement.

FAQ

Is structured output the same as JSON mode?

No. JSON mode guarantees valid JSON; structured outputs guarantee your schema.

Which models support it?

Structured outputs are available from GPT-4o onward.

Can I combine it with function calling?

Yes. Use tools with strict: true and tool_choice set to a specific function.

Do I still need validation after parsing?

Yes. The schema covers shape, not business logic or truthfulness.

What if the model refuses?

Refusals are detectable and structured. Log them and respond gracefully instead of retrying blindly.

Related Articles

Nov 17, 2026

AI Medical Notetakers: Balancing Efficiency with Human Care

Discover how AI-powered ambient scribes are transforming clinical documentation while preserving the essential human touch in modern healthcare.

Nov 17, 2026

AI Agents: How They Work and the Security Risks to Manage

AI agents plan, act, and adapt with minimal supervision. Learn how they work, the security risks they introduce, and how teams can use them responsibly.

Nov 17, 2026

How AI Travel Assistants Are Redefining Trip Planning

Specialized AI travel assistants build itineraries, rebook disrupted trips, and learn traveler preferences. A look at what these agents can and cannot do.