JSON Mode Ships. Schema Correctness Still Slips.
Constrained decoding makes malformed JSON impossible. It does nothing to make the JSON true — and that gap is where production quietly breaks.
Use it — but only for the syntax. Grammar-constrained decoding makes malformed JSON structurally impossible, which kills retry loops and regex scraping in production. It does not make the output correct: models still fabricate required fields and emit valid-but-wrong values. Treat schema-validity as necessary, never sufficient, and keep a semantic gate.
The Weights Desk · 4 min read- Grammar-constrained decoding guarantees parseable JSON by masking illegal tokens — syntactic failure becomes structurally impossible.
- Valid JSON is not correct JSON: required-field pressure makes models fabricate plausible values instead of signaling uncertainty.
- Forcing structure before reasoning can flatten output quality; split the reasoning pass from the extraction pass.
- HELM's lesson transfers: one metric misleads, so judge structured output on semantics and calibration, not merely whether it parses.
- Ship it for the syntactic layer; add a downstream semantic validation gate before you trust any single field.
Verdict: ship it for syntax, gate it for truth
Here is the call: use schema-constrained decoding in production, and treat a valid parse as the floor, not the finish line. The win is real. When the decoder masks every token that would break your grammar, malformed JSON stops being a failure mode you handle and becomes one that cannot occur. Retry loops shrink. Regex-scraping the model's prose disappears. That part ships. The caveat is equally real, and it is where most teams get burned: a well-formed object can still be wrong. The envelope is guaranteed; the letter inside is not.
What made it ship: malformed JSON became impossible
The mechanism is unglamorous, and that is the point. Grammar- or FSM-based decoding restricts the sampler at each step to tokens the schema permits. Structural validity is enforced by construction, not hoped for after the fact. Vendor tool-calling and schema response modes push the same guarantee up to the API. Before this, 'the model returned JSON wrapped in an apology' was a daily incident. Now it is not. If you own the decoder, or your provider exposes a strict schema mode, wire it in and delete the brittle parser you built around the old failures.
What would have killed it: mistaking valid for correct
What kills the pattern is trusting the parse. Constrained decoding does nothing for semantics. Under a required field, a model will manufacture a plausible value rather than admit it does not know — a fail-open behavior that is dangerous precisely because the output looks clean. HELM's core argument lands here: a single metric misleads, and models must be measured across distinct axes, calibration and robustness among them, not accuracy alone (s1). Read 'did it parse' as one axis, never the verdict. The fix that worked in practice: split a reasoning pass from an extraction pass so structure never front-runs thinking, and put a semantic validation gate downstream. Force the schema too early and you can flatten the reasoning that made the answer right — untested at scale across every model, but a repeatable failure in the ones I have watched. Ship the syntax. Verify the meaning.
- Does JSON mode guarantee my output matches my schema?
- Only the syntactic envelope. Basic JSON mode guarantees well-formed JSON; full schema adherence needs grammar/FSM-constrained decoding, and even that guarantees structure, not semantic truth.
- Can constrained decoding hurt answer quality?
- Yes. Forcing the token distribution into a schema before the model reasons can flatten chain-of-thought. Let it reason first, then extract into the schema.
- So do I trust a valid parse?
- Treat it as one axis, not the verdict. A clean parse can wrap fabricated required fields, so pair it with a downstream semantic check.