Structured Output Error Rates: Schema-Valid vs Correct
"Structured output error rate" and "accuracy" measure two completely different things, and vendor marketing routinely lets them blur together. A structured-output error is a format failure — malformed JSON, a value outside the allowed schema, an unparseable field. An accuracy error is a content failure — the output is well-formed but wrong. TypeSafe AI's launch claims for Jev make both numbers available side by side, which makes this a good case study in why you need both, not just one.
Quick answer: Schema-valid means the output has the right shape (right type, right field, value from the allowed set). Correct means the value is actually the right answer. TypeSafe AI reports Jev has a 0% structured-output error rate (every output is schema-valid, by design — it can't fall outside the schema) and 67.8% accuracy on its own four-workflow benchmark (roughly a third of answers are wrong, despite being perfectly formatted). Both numbers are real and both matter, but they answer different questions — and the "0%" one is much easier to satisfy.
What is a structured-output error, specifically?
It's a failure of format, not content: invalid JSON, a missing required field, a value that isn't a member of the defined category list, a type mismatch (a string where a number was required). For a chat LLM constrained by JSON mode or function calling, this can still happen at the edges — a truncated response, a schema violation the constraint didn't fully prevent. For Jev, TypeSafe AI claims this class of error is eliminated entirely (0%) because the architecture doesn't generate a string that could fail to parse in the first place — it emits a typed value directly, constrained by construction rather than by a post-hoc check. See typed decisions vs JSON mode for how that differs mechanically from constrained decoding.
What is accuracy measuring instead?
Whether the schema-valid output is the right one. A Choice question can correctly return a value from the allowed category list and still pick the wrong category. A Score can be a valid number on the right scale and still be a bad score. A Noul can be a valid probability between 0 and 1 and still be poorly calibrated or flatly wrong about the likely outcome. Accuracy (and separately, calibration — see calibrated confidence scores) measures this, and it requires ground truth to check against, not just a schema validator.
Why is this distinction the most important thing to get right about Jev?
Because "0% structured-output error rate" sounds, at a glance, like "never wrong." It isn't. TypeSafe AI's own four-workflow benchmark (security incident response, observability, invoice processing, customer service) puts Jev's accuracy at 67.8% — meaning close to a third of its answers on that benchmark were incorrect, even though 100% of them were schema-valid. The honest way to describe this: Jev can't hallucinate a malformed answer, but it absolutely can produce a confidently wrong, perfectly-typed one. "Can't hallucinate" here is a narrow, format-level claim, not a correctness guarantee.
| Metric | What it measures | Jev, as reported by TypeSafe AI |
|---|---|---|
| Structured-output error rate | Is the output the right shape/type? | 0% (vendor-reported, by architecture) |
| Accuracy (4-workflow benchmark) | Is the output the right value? | 67.8% (vendor-reported, not independently reproduced) |
How do I measure both for any model I'm evaluating?
For format validity: run every response through a schema validator and count failures — this is mechanical and objective. For correctness: you need labeled ground-truth examples (or a reliable proxy, like downstream outcome data) and you compare the model's answer against the true answer, computing accuracy per question type (Choice, Score, Noul each need slightly different accuracy metrics — exact match for Choice, error/rubric distance for Score, calibration curves for Noul). Neither number substitutes for the other, and a vendor quoting only one of them is telling you half the story.
Common pitfalls
A few mistakes come up repeatedly when interpreting these numbers:
- Reading "0% error rate" as "0% wrong" — it's a format claim, not a correctness claim.
- Reporting only a structured-output error rate in a vendor pitch without an accuracy number alongside it.
- Trusting a vendor-reported accuracy figure (like Jev's 67.8% on TypeSafe's own benchmark) without independent reproduction or testing on your own data.
- Assuming a low structured-output error rate reduces the need to monitor for wrong-but-valid answers in production.
Pro Tip: Whenever you see a "hallucination-free" or "0% error rate" claim about any structured-output model — Jev included — ask immediately: error rate on what, format or content? The two questions have very different answers, and only one of them tells you whether you can trust the decision.
← Back to AI & ML Tips