LLM Routing Patterns: Cheap Classifier, Then Escalate

The most common cost- and latency-saving pattern in production LLM systems is routing: run a cheap, fast classifier first, and only call an expensive frontier model for the fraction of requests that actually need it. TypeSafe AI positions its new System One model, Jev, explicitly for this role — "a fast decision layer that classifies, scores, and routes," escalating to frontier LLMs for open-ended work — but the pattern itself predates Jev and works with any fast classifier, typed-decision model or otherwise.

What does a routing layer actually do?

It classifies each incoming request before deciding how to handle it — often into a small set of buckets like "auto-resolve," "route to specialist model," or "escalate to frontier LLM / human." The classifier's job is narrow and repeated (the same decision, over and over, at volume), which is exactly the profile TypeSafe AI describes Jev's Choice and Noul question types as suited for: see Jev question types for the mechanics, and Jev vs LLM for when a typed decision is the right tool for this step.

What's the cost math behind routing?

TypeSafe AI's own pricing for Jev: input at $0.042 per million tokens, output tokens free/unmetered, and a reported per-case cost around $0.0004 versus $0.0304–$0.1761 for frontier LLMs on their four-workflow benchmark. Those are vendor-reported figures for TypeSafe's specific benchmark tasks, not a general result — but the shape of the math is the standard routing argument regardless of which fast classifier you use:

# Illustrative cost math, not vendor-verified for your workload
total_requests = 100_000
pct_routable   = 0.85          # handled by fast classifier alone
pct_escalated  = 0.15          # needs frontier LLM

cost_fast_per_case     = 0.0004   # vendor-reported, Jev, benchmark-specific
cost_frontier_per_case = 0.03     # varies widely by model/task

cost_with_routing = (total_requests * cost_fast_per_case) + \
                     (total_requests * pct_escalated * cost_frontier_per_case)
cost_all_frontier  =  total_requests * cost_frontier_per_case

# cost_with_routing is dramatically lower whenever pct_escalated is small

The bigger the fraction of traffic that's routine and well-defined, the more routing saves — both in dollars and in latency, since most requests never wait on a multi-second frontier-model call.

How do I decide the routing threshold?

Off the fast classifier's confidence, if it's calibrated — route anything above a high-confidence threshold to auto-handling, send low-confidence or out-of-schema cases to the frontier model. This only works if the confidence score is genuinely calibrated, not just monotonic; see calibrated confidence scores for how to check that before trusting a threshold in production. An uncalibrated "confidence" can make your routing threshold arbitrary rather than meaningful.

What's the risk of routing on a fast classifier's mistakes?

The classifier can be confidently wrong in a well-typed way — it returns a schema-valid answer that's still incorrect. On TypeSafe's own four-workflow benchmark, Jev scored 67.8% accuracy, meaning roughly a third of its answers were wrong even though 100% were structurally valid (its reported 0% structured-output error rate is about format, not correctness — see structured-output error rates). A routing layer built on any fast classifier needs monitoring for misrouted cases, not just a one-time accuracy check at launch.

Common pitfalls

A few mistakes come up repeatedly when building a routing layer:

• Routing purelyon speed/cost savings without measuring how often the fast tier's decisions are actually correct.
• Setting a confidence threshold before verifying the confidence score is calibrated on your data.
• Never re-evaluating routing accuracy after launch — routing decisions compound quietly when they're wrong.
• Applying vendor-reported cost/accuracy ratios from someone else's benchmark tasks directly to your own workload without testing.