Structured Outputs: Connecting Probabilistic Text to Deterministic Software Contracts

When people understand a passage, they rely on context and intent. If a message adds an 'OK', omits a quote mark, or uses a full-width comma, as long as the meaning is unchanged, people hardly notice. Programs are exactly the opposite: they do not guess intent but compare character by character according to predeclared rules. If the caller expects an object but receives a string, the program crashes at type checking; if a field is named userName but the contract says username, the downstream logic gets a null value. The fault tolerance of natural language comes from leniency under shared context; the fault tolerance of programs comes from precision at boundaries: as long as boundaries are declared clearly enough, anomalies can be mechanically identified and rejected without a person guessing 'what the model probably means this time.'

Structured output is exactly the transformation of the former communication style into the latter. Its input consists of two parts: task semantics—that is, 'what question this call should answer and what content it should return'—and a versioned data contract—that is, 'what shape the answer must take.' Its output is not free text but an object that is parseable, validatable, and rejectable by programs at the field level. Generation constraints and parsers together complete this conversion: they turn the fault tolerance that natural language resolves through context into explicit fields and explicit states—whether a field is present, whether the type is correct, and whether the value falls within the allowed range; each item has a definite answer.

This transformation materializes as a chain that tightens step by step. Writing 'return JSON' in the prompt only increases the probability that the output is JSON; the model may still add comments, add explanatory text, or not return JSON at all. JSON mode usually preserves syntax: the output is constrained to be legal JSON text. But legal JSON only guarantees that a machine can parse it, not that the content is usable—fields can be missing, types can be wrong, and values can exceed allowed ranges. JSON Schema further constrains shape above this layer, declaring fields, required items, enum values, length limits, and version numbers, so that consumers mechanically reject anomalous objects instead of using fragile regular expressions to guess the model's intent. The end of the chain is constrained decoding: it blocks illegal paths during generation, so that token sequences that do not conform to the contract never have a chance to be sampled. The four layers tighten in increasing order, but the support range of each layer varies by implementation, so no matter which layer the upstream claims to support, the receiving side must still perform server-side revalidation.

Here we need to separate 'format validity' from 'content trustworthiness.' An object that passes all syntax and shape validation only shows that a machine can read it and locate the agreed fields; it does not show that the values in the fields are true, nor that execution is allowed. The contract turns the answers to validation questions into Boolean values, but the model's judgment about facts is still probabilistic. The value of a data contract is precisely to make failures explicit, local, and countable: which call, which field, and which constraint was violated can all be recorded and located, rather than letting errors melt into a long passage of natural language like typos that cannot be traced afterward. It constrains the output form, not the model's knowledge reliability; it allows probabilistic models to be safely consumed by deterministic software, but it does not turn probabilistic models into trusted databases.

Being parseable by JSON.parse is still far from "executing a refund". Successful parsing only proves that this text is valid JSON; it cannot answer any business question: whether the fields are present, whether the values are reasonable, whether the requester is authorized. To turn a model-generated string into an actually executed refund, it must pass through four gates, and they must be passed in order — passing an earlier gate cannot substitute for a later one, and if any gate fails, downstream should not "keep going by guessing as best it can".

Layered validation takes four inputs: the object returned by the model, the agreed-upon schema, authoritative business data (from the database or server-side state), and the subject of the current request (who is initiating this operation). The output is one of five verdicts: pass, field error, need more information, reject, or escalate. Each of the four gates checks a different kind of problem:

The first gate only makes a purely syntactic judgment: brackets pair, quotes close, escapes are legal, and the text ends at the correct position. Model output is often truncated or interleaved with explanatory text; this layer keeps out any input that is "not complete JSON", and on failure it rejects the whole message or triggers regeneration.

The second gate checks shape against the schema: whether required fields are present, types match, enum values are within the allowed set, and numbers fall within declared ranges. The action field allows only the two values refund and ask; if the model writes any other string, the error is precisely located at the field level, rather than letting downstream code continue running with an erroneous value.

The third gate deals with business facts. A correct schema does not mean the content is correct: the amount field can be a number and yet exceed the order's actual refundable limit. This layer checks cross-field rules (each of two fields may be individually valid but contradictory in combination), queries the database to verify state, and requires evidential support; in the example, that means amount ≤ order refundable amount. On failure, the system should ask the user for more information or transfer to a human, rather than guessing a "plausible-looking" value on its own.

The fourth gate handles permissions and side effects: execution will change real-world state, so it must confirm who the subject is, whether policy allows it, whether the request is idempotent, and whether approval is required. The question to answer in the example is "can the current customer service agent approve this refund". Failure means prohibiting execution or escalating to someone with authority.

The four gates each answer a question downstream cannot answer on its behalf, and they must be passed in order: the syntax layer cannot judge field validity for the schema layer, the schema layer cannot verify facts for the business layer, and the business layer cannot decide who may execute for the permission layer. A failure at any layer stops the flow at that layer, explicitly rejecting or escalating. It is worth emphasizing that the user_id, amount, and action supplied by the model are all untrusted suggestions: they are merely structured text, and do not gain any trustworthiness from that. The execution layer must re-obtain the subject and permissions from the authentication context, and must not use the identity claimed by the model to query the database or authorize.

Cramming every field that might be used into one nested object looks like a one-shot solution, but it actually raises both the error rate and coupling: every additional optional field adds a batch of “legal but meaningless” combinations; any change to a field can affect the validation of other fields, consumers’ parsing, and future migration. The minimal schema approach is the opposite: the input consists of the fields the business genuinely needs the model to judge, the facts the server already knows, and compatibility requirements; the output contains only the fields the model must decide and explicit branches, and everything else is not handed to the model.

The first principle of splitting is: only let the model produce content that requires semantic judgment. Timestamps, user IDs, totals, and derived state are values the server already has in the request context; if the model repeats them, it may copy them incorrectly, so the server fills them in. The model is responsible only for its unique contribution—the intent, parameters, and reasons inferred from the user's language. Identity is filled in by the server, not from what the model claims, which is consistent with the conclusion of the validation layer.

The second principle of splitting is to use enums and discriminated unions to shrink the state space. Do not use free-form strings for action names: an enum saves consumers from guessing spellings, but it also incurs a downstream cost—new enum values require compatibility handling, and old consumers that encounter a new value must be able to degrade safely. Mutually exclusive tasks should be expressed as discriminated unions: first use a kind field to choose between refund and ask, then apply the corresponding small schema. Each of the two small schemas declares only the fields it needs, making it easier to constrain, validate, and migrate than one large object containing dozens of optional fields, because invalid combinations cannot be expressed in the first place.

Field declarations themselves also have a set of tightening rules: distinguish missing from null—a missing field and an explicit null are two different states, and conflating them loses information; limit the length of strings and arrays to block resource exhaustion and absurdly long values; reject additionalProperties by default so that unknown fields directly expose schema drift instead of being silently swallowed. Every rule's benefit corresponds to a risk that must be managed:

Required fields guarantee state completeness, at the cost that when facing an unknown value you cannot take the easy way out by filling in a fake default—rather, report an error explicitly or ask; length and range limits block extreme input, but boundary values must be updated in sync with business rules, otherwise the validator will reject legitimate real requests; forbidding extra fields makes differences between old and new versions immediately visible, at the cost that every change requires negotiation between provider and consumer.

Taken together these rules point to the same conclusion: the smaller the schema, the fewer choices the model has to make, and each declaration corresponds to a real business decision; the larger the schema, the more declarations become empty words, and the error rate balloons along with the number of paths. At the same time, each field should be accompanied by an explanation of its business meaning and positive and negative examples, so that error messages on validation failure can tell a person “what is missing and why it is wrong,” rather than just printing a stack trace.

The model returns action=refund and amount=120. This output passes the syntax gate—it is complete, valid JSON—and also passes the schema gate—action is within the allowed enum, and amount falls within the general range 0–500 declared by the schema. Both gates let it through, yet it still cannot be executed. The problem lies in the next two gates, and what they check is no longer the model output itself.

The case has four inputs: a structurally valid object, the order’s paid and refunded records, the policy amount cap, and the current operator’s role. The output is a business error code or a safe action. The business facts gate must verify an order invariant: the refundable amount R equals the smaller of the paid amount P minus the already-refunded amount D and the policy cap L, that is,

R = min(P − D, L)

Here P is the order’s paid amount, D is the amount previously refunded, L is the policy-allowed amount cap, and min means taking the smaller of the two candidate caps. Substituting this order’s data, the maximum refundable amount is 80. 120 exceeds 80, violating the order invariant. Note that this judgment depends on authoritative order and policy data; the schema’s general range cannot replace it: 120 within 0–500 has no shape problem at all—the nature of the problem is completely different. The permission gate then gives a second rejection reason: the current role’s approval authority does not cover this amount, and there is no approval right.

The illustration draws this path as a flow in which the refund structured output passes sequentially through four validation gates: syntax, schema, business facts, and permission. The core relationship conveyed by the figure is: format guarantees only make errors easier to detect; the facts in the database and the current actor’s permissions ultimately decide whether execution can proceed. The first two gates are checks of the model output’s conformance to the contract; the last two gates check the output against the real world.

The correct recovery after failure is to return the business error code amount_exceeds_remaining, so that the model, based on this, switches to an explanation or routes to a human, rather than 'quietly handling it': silently truncating 120 to 80 and then executing would amount to the system arbitrarily rewriting the model’s intent—the model may indeed think a refund of 120 is due, truncation hides an upstream problem somewhere, and it also changes 'the system detected a violation' into 'the system executed another refund according to its own understanding.' Better to let the flow stop in place with an explicit error than to repair a validation-failed object with guesswork.

Since the server will eventually validate, why constrain at generation time? Because generation-time constraints and server-side validation guard against two different kinds of errors and save two different kinds of costs. The inputs to this two-layer mechanism are the model distribution, the schema, and server-side business rules, and the output is an object whose format is constrained and that has been verified after generation: constrained decoding reduces syntax- and schema-level invalid outputs, thereby reducing futile retries; post-generation validation guards against implementation defects, version mismatches, and semantic errors—problems that only appear when the decoder's promises do not match reality and are invisible at generation time.

The mechanism of constrained decoding is constrained sampling: at each step, set the probabilities of illegal tokens in the current state to zero, then renormalize the remaining legal tokens. Written as a formula, the constrained probability P′(t|s) of token t in state s is proportional to the original probability P(t|s) multiplied by an indicator function:

P′(t|s) ∝ P(t|s) × 1[t ∈ M(s)]

where P(t|s) is the probability of the original token t in state s, P′(t|s) is the probability after applying the constraint, M(s) is the set of tokens that are still legal at the current step, 1[·] is the indicator function that takes 1 for legal and 0 for illegal, and the symbol ∝ indicates that the right-hand side must be divided by the total probability of all legal candidates to be normalized into a probability distribution.

This formula also illustrates the boundaries of constraints. The indicator function merely cuts off the probabilities of illegal candidates and proportionally amplifies the remaining ones; it does not add any new information. If the model originally placed most of its probability on illegal candidates, after normalization it can only be forced to choose legal values with very low probability—the format passes, but the content may become worse. Therefore, we cannot look only at structural pass rate: a format success but a decline in business quality is precisely the constraint pushing distribution problems to the surface. The correct metrics must simultaneously consider first-try structural pass rate, business correctness, and task quality; all three are indispensable.

Precisely because the two layers each handle their own part, post-generation validation cannot be omitted. The underlying API's support for schema features varies by implementation, and claiming support does not mean real support: certain nested structures and certain keywords may be ignored at runtime. A prudent approach is to run contract tests at startup to verify the claim and re-validate after each generation, rather than trusting the documentation by default. The accompanying principle is not to use automatic repair to mask distribution problems: a model that 'fixes JSON' may well change the meaning of fields while adding quotation marks. Only purely syntactic repairs that can be proven equivalent to the original meaning—such as completing truncated closing brackets—are suitable for automatic application; any 'repair' involving field values or types must return to the explicit failure-and-retry flow.

After a validation failure, the worst approach is to feed the entire error object back to the model verbatim for unlimited retries. This simultaneously raises latency and cost, repeatedly sends untrusted text back into the context, expands the prompt injection surface, and, more dangerously, every retry may be treated as a new action and executed again. Failure recovery must be local, bounded, and side-effect-free; all three are indispensable.

The input to the recovery process is not the raw error message, but machine-generated field paths, error codes, allowed ranges, and the request's idempotency key; the output is one of five actions: localized repair, user clarification, fact refetching, rejection, or human escalation. Among these, there are explicit rules for 'what to echo back to the model': only pass back machine-generated error information—which field violated which constraint, and what the allowed values are—not the large amount of untrusted text the model itself produced. The error object may contain user-injected content, and passing it back verbatim to the model gives the injection a direct channel.

Side-effect-freedom is a hard temporal constraint: absolutely no side effects are performed before the first validation passes. Validation is essentially a pure read operation, so it can be retried for free; once a side effect occurs, it cannot be undone for free. Retries must reuse the request ID and idempotency key, ensuring that no matter how many times the same logical request is retried, the business system recognizes it only once. Being bounded is a hard constraint on the number of attempts: the number of retries must be capped, and after consecutive failures of the same kind, one must not keep blindly retrying; instead, downgrade to a simpler schema or a deterministic form—making the task's difficulty decrease as the model continues to fail, rather than letting failure loop indefinitely.

Only by dispatching actions according to the cause of failure can recovery be called local. For pure syntax failures, such as truncated output or a missing closing brace, handle with constrained regeneration or a one-time repair. When a required field is missing and the user can be asked, return ask to give the problem back to the user rather than fabricating a value on the user's behalf. When there is a factual or business conflict, refetch authoritative data to verify; do not let the model guess. When permissions are insufficient, prohibit execution and escalate to a human if necessary. Each category of failure stays in its own layer and does not spread into a restart of the entire task.

Finally, the operation of this mechanism must be supported by metrics: record the first-pass rate, per-field errors, post-repair pass rate, final task failure rate, number of retries, and additional latency introduced by retries (p95). Without these numbers, local, bounded, and side-effect-free are just slogans—only by seeing which field errors cluster in and which layer retries concentrate in can we know whether the schema is declared too strictly, or whether the model simply should not be given the responsibility of direct generation for this task.

Why might old consumers silently misinterpret when the producer upgrades the schema? Because the contract is not updated simultaneously by both parties sitting down together: producers and consumers deploy independently, so version interleaving inevitably occurs in between. When an old consumer parses a new object, it may simply discard newly added fields, or stuff newly added enum values into the wrong default branch—no error is reported, but incorrect logic is executed. The inputs to contract migration are the old schema, the new schema, the respective versions of producers and consumers, and historical trace; the outputs are a compatibility matrix and an expand/contract release plan.

The first trap in judging compatibility is thinking that “additive equals compatible.” Adding optional fields is usually backward compatible, provided that consumers do not reject extra fields—a strict consumer that rejects additionalProperties by default will fail immediately when it encounters a new field, and this is no longer compatible. Adding enum values is equally dangerous: an old consumer’s switch statement does not recognize the new value and falls into the default branch, and the default branch often contains an incorrect fallback action. Truly breaking changes are more direct: deleting fields and changing field types break both the read and write ends. Tightening a range is also a breaking change: reducing the upper limit of an amount from 0–500 to 0–200 makes old data no longer valid.

Therefore, the schema must carry a version number and a clear compatibility policy, and releases use the expand/contract three-step process: first expand—let consumers accept both old and new formats simultaneously; then switch—let producers start sending the new format; finally contract—retire the old format. The risks and migration paths for each type of change are as follows:

For adding optional fields, first update all consumers to be able to accept the field, and then have producers output it. At the same time as adding enum values, all consumers must add an explicit unknown branch to ensure that encountering an unrecognized value does not result in “defaulting to normal execution.” Renaming or changing the type of a field requires a transition period of dual-write and dual-read: let both formats coexist for a period of time before retiring the old field. Tightening a range requires first migrating existing data into the new range, and validation logic must be executed separately according to the version to which the data belongs, rather than using new rules to judge old data.

Version belongs not only to the schema file alone: prompts, schema, constraint compiler, and consumer versions must be recorded together. When a problem occurs, only by restoring these items to the combination at the time and replaying old trace can you reproduce the rules and behavior of that time; looking only at today’s prompts and today’s schema can never explain yesterday’s errors.

Schema already restricts action to refund and ask; why can prompt injection still cause harm? Because the attacker's goal is never to make the model output invalid JSON, but to induce it to choose parameters that are syntactically legal but harmful in content: refund is a legal action, but the object can be someone else's order, the amount can hit the upper limit, and the reason can hide instructions for a downstream agent. Structured constraints weld shut the opening of 'illegal format', leaving the opening of 'legal format but hijacked intent'.

The inputs to secure execution are valid parameters, authenticated principal, tool whitelist, policy and approval status; the output has only two possibilities: execute with least privilege, or reject. The first principle is that three things provided by the model are all untrusted: identity, tool name, free text. The server obtains user/tenant from session authentication and does not accept identity self-reported by the model; the tool registry only exposes capabilities allowed by the current flow. When the model 'calls out' a tool name, it only means it suggested that name; which real capability it ultimately maps to is determined by the registry and policies. Every parameter must pass object-level authorization, rate, amount, purpose, and approval checks. High-risk actions are first planned by the model, then go through deterministic policy gates and human confirmation before submission — plans and suggestions can come from the model, but the decision authority must rest with deterministic code or a human.

Free-text fields are the most easily underestimated channel in structured objects. Attack payloads can be placed without breaking the schema: SQL snippets, HTML snippets, or instructions written for the next agent are completely legal when placed into a string field. Structure does not equal sanitization; it just puts the attack payload into a valid string. Any consumer must escape and mark these fields as untrusted according to the target context before using them — escape before rendering as HTML, parameterize before concatenating into SQL, and isolate before passing to the next agent.

Together these rules prevent the confused deputy problem: the model proposes actions on behalf of the user, but it must not inherit all of the service account's permissions. If the execution layer takes the model's output and directly calls a fully privileged internal interface, the model becomes a manipulated deputy, and the injector borrows far greater permissions than the user through it. Therefore every tool call must be re-authorized with least privilege and the current principal — not 'this round was already authorized', but each action, each object, and each call is independently verified. Identity claims, tool selections, and free text output by the model must all be treated as untrusted data, so that the security boundary is not bypassed along with a legitimate refund object.

“Structured Outputs success rate 99.9%” sounds close to perfect, but it doesn’t answer the most critical question before launch: which layer the 0.1% of failures occurred in, and what consequences they caused. This number is likely to average “a format error that was retried once” and “an unauthorized refund” in the same denominator, and the cost of the latter is far beyond the order of magnitude of the former. The inputs to end-to-end evaluation are model outputs, validation logs, tool results, and business consequences; the outputs are metrics that unfold along the entire contract: syntax, schema, fields, facts, permissions, idempotency, task quality, latency, and cost.

Reporting by layer means each gate has its own number: full syntax rate, schema first-pass rate, dead-end rate (requests that exhausted retries), field-level error distribution, business/factual pass rate, permission interception count, tool success count, repeated side-effect count, final task quality, additional latency, and cost. A request blocked at the syntax gate is an entirely different kind of failure from one blocked at the permission gate; mixing them together loses the ability to locate the problem. Regression sets should specifically cover boundary and adversarial scenarios: truncated output, Unicode text, extra-long arrays, unknown enum values, legacy consumers, existing data after new policies take effect, and malicious free text.

Error budgets must be graded by final consequence. A large number of harmless format retries and one unauthorized refund cannot be placed in the same average failure rate and cancel each other out: the former is only a cost issue, while the latter is a security incident. The safe approach is to set independent hard thresholds for high-risk categories—the acceptable number of unauthorized-action errors is zero or near zero, with immediate rollback—rather than “overall 99.9% compliance”.

The diagnostic order is also determined by the contract structure: first ask whether the output is complete, then whether the shape is correct, then whether the facts and relationships hold, and finally whether the current subject can execute safely. The order of these four questions corresponds to a troubleshooting path from cheap to expensive, from format to consequence. Following this order, any failure can be classified into the layer where it actually occurred, rather than being vaguely recorded as “the model output was wrong”; only in this way can the remaining 0.1% of the “99.9% success rate” be decomposed into a concrete, fixable, consequence-prioritized list of problems.

The original problem that structured output solves is the gap between the fault tolerance of natural language and the determinism required by programs. Starting from this gap, each design decision is motivated by the shortcomings of the previous one, leading all the way to verifiable practices; the entire causal chain is composed of six steps.

The starting point is a minimal versioned schema. It reduces “what the model must decide” to only the fields that represent real business decisions, leaving identity, time, and derived values to the server; at the same time it carries a version number, providing a coordinate for future expand/contract evolution. The smaller the schema, the smaller the state space on both the generation and validation ends.

The second step is to constrain generation and wait for a complete acceptance state. Constrained decoding masks illegal tokens during sampling, reducing ineffective retries; “waiting for a complete acceptance state” means not treating half-finished text from streaming output as a result—only when the generator gives a complete end signal does the object proceed to the next step.

The third step is for the server to reparse and perform schema validation. The server does not trust upstream self-declarations; it reparses the data and checks fields, types, enumerations, and ranges against the schema, keeping implementation defects and version mismatches out. At this point the format is machine-confirmed to be readable, but that still does not mean the content is trustworthy.

The fourth step is to validate facts and cross-field business invariants. Here database facts and policy data are introduced to check rules such as R = min(P − D, L) that depend on authoritative data; failures return business error codes rather than silently correcting the value.

The fifth step is to execute idempotently after authenticating the current principal. Identity comes from session authentication rather than model self-reporting; tool calls are reauthorized with least privilege and the current principal; execution carries an idempotency key, ensuring retries do not cause duplicate side effects.

The sixth step is to record by failure layer and perform limited recovery or rollback. Each layer has its own failure modes and recovery actions: syntax failures are handled by constrained regeneration or a one-time repair; missing required fields turn to ask_clarification; business conflicts refetch authoritative data; insufficient permission is denied or escalated; repeated failures of the same kind degrade the schema. Recovery is local, limited, and side-effect-free, and each layer's pass rate, errors, and latency are recorded, becoming evaluation data for locating failures along the entire contract.

The causal direction of the chain is one-way: schema constrains generation, generation constrains server-side parsing, parsing leads to business validation, business validation leads to authenticated execution, and execution results in turn become input for the next round of evaluation and version evolution. If any link bypasses the previous one—executing directly while skipping the schema, truncating amounts directly while skipping business validation, calling tools directly while skipping authentication—the chain breaks at that point. This concept starts from the observation that programs and natural language tolerate errors differently, and ultimately leads not to a particular model parameter but to a whole engineering structure that can be mechanically verified, located by layer, and recovered in a limited way.