An AI answer can be perfectly formatted and still be wrong. And a correct answer can be useless just because it came back in the wrong shape. Anyone who puts AI in charge of feeding a system — an ERP, an order spreadsheet, a support queue — learns this fast: the problem isn’t that the model writes badly, it’s that the model writes text when the system on the other side needed a filled-in form.
That is what structured output is for: instead of asking “please answer in JSON” and hoping, the company hands the model a schema (a JSON Schema, with the fields, types and allowed values) and generation itself is constrained to it. Understanding how this works under the hood changes what you should expect from the technique — and, more importantly, what you still need to check.
The problem: free text doesn’t feed a system
Imagine a services company that receives 300 quote-request emails a week. The idea is simple: AI reads each email and extracts the customer, the service requested, the deadline and the urgency, which go straight into the sales system. In a test with 20 emails, it works. In production, one in every fifty answers comes with a sentence before the JSON (“Sure! Here is the extraction:”), another swaps “urgent” for “high priority”, another returns the date as “next week”. Each of these breaks the integration — or, worse, lands in the system with a value nobody anticipated.
The instinct is to strengthen the prompt: “answer ONLY in JSON”, “use only values X, Y and Z”. That reduces errors but doesn’t eliminate them, because a prompt is an instruction, not a constraint. The model is still free to generate any sequence of text; it’s just more likely to follow the instruction.
How it works under the hood: constrained decoding
A language model generates its answer one token at a time. At each step, it computes a probability for every possible token in its vocabulary and picks one. With structured output, the schema is compiled into a grammar — a set of rules that says, at each point in the answer, which tokens are valid. Before every choice, tokens that would violate the schema get zero probability. If the next piece must be a field name, only field names from the schema can come out. If the field is a closed list of values, only those values can come out.
The difference in reliability is large. When OpenAI launched the feature in August 2024, it reported that, in an internal eval with complex JSON schemas, the model with structured outputs followed the schema 100% of the time, versus less than 40% for an earlier model relying on prompt instructions alone. The figure comes from the vendor, on its own test — but the mechanism explains why it makes sense: the model is no longer persuaded to follow the format, it is prevented from leaving it. Today the major model providers all offer some version of this feature.
There is one technical limit worth knowing: if generation is cut off midway (by a token limit, for example), the answer can arrive incomplete. The guarantee covers the path, not the arrival.
What the format guarantees — and what it doesn’t
Here is the central distinction: structured output guarantees form, not content. The “deadline” field will always exist and will always be a valid date. Whether that date is the one the customer actually asked for, the schema has no idea.
And there is a side effect that rarely gets discussed. If the schema declares “deadline” as required and the email mentions no deadline at all, the model has no way to say so — it is forced to produce a date. And it will produce a plausible one. A poorly designed schema doesn’t just fail to prevent fabrication: it demands it. A rigid format without an honest way out is a factory of made-up data that looks like reliable data.
When to use it: three questions before deciding
Not every AI task needs structured output. A summary for a manager to read, a reply to a customer, a draft proposal — that is text for people, and forcing a schema only makes it poorer. The three questions that separate the cases:
- Will the answer be read by a person or by a system? If the destination is other software (ERP, CRM, spreadsheet, API), structure stops being a preference and becomes a requirement.
- Does the answer fit into closed categories? Classifying urgency, industry, request type, yes/no — when the options are known, a fixed list of values eliminates the vocabulary drift that breaks reports and automation.
- Is the volume too large for someone to check one by one? With 20 cases, a person fixes the format by eye. With 300 a week, a format error needs to be impossible, not rare.
Two “yes” answers already justify a schema. None, and free text is probably the right choice.
Three design decisions that change answer quality
Once you choose a schema, its design matters as much as the model. Three decisions make the biggest difference:
1. Field order. Because the model generates one token after another, whatever comes first in the answer is decided before whatever comes after. A schema that asks for “urgency” in the first field and “email excerpt that justifies it” in the last forces the model to decide before looking at the evidence. Flip it: evidence first, conclusion after.
2. An honest way out. Every field that may have no answer in the source text needs to accept “not provided” (or empty). That turns the pressure to invent into a truthful answer — and it’s measurable: how many requests arrive without a deadline is, in itself, useful data for the sales team.
3. Business-rule validation after the AI. The schema checks type and format; the business rule is still yours. A deadline in the past, a value outside the normal range, a customer who doesn’t exist in the records — those are checked afterwards, with simple code, and whatever fails goes to human review instead of straight into the system.
The schema is the business rule, written down
There is an effect that goes beyond the technique. To write a good schema, someone has to decide which request types the company handles, what counts as urgent, which fields are mandatory for a quote to move forward. In many companies, none of this is written down — it lives in the head of whoever has done the triage for years.
As João Paulo Batistella, an innovation executive, argues, AI depends on process formalized in data, not on knowledge that exists only in someone’s head — and before any agent, you have to define what the AI can and, above all, cannot do. The output schema is exactly that, in code: the process rule, written in a way that neither the person nor the machine can work around. When a team gets stuck writing it, the problem is rarely technical. It’s a sign the process wasn’t defined yet.
The practical test is simple: take the AI task that feeds another system the most today and ask whether anyone in the company can write, on a single sheet, the fields it should fill and the values allowed in each. If they can, structured output is an afternoon’s adjustment. If they can’t, the first task isn’t a technology task.
For more analysis on how AI works under the hood and what it changes in management, follow @ElevaTechno on X, @elevatechnologies on Instagram or Eleva Technologies on LinkedIn, or learn more about the group at elevatec.net/about.
