The Routing Problem: Architecting Boundaries for Execution

Andrew Siebert

04 Sep 2026 • 6 min read

The Context Gap ended on an unresolved handoff: closing the gap gets the right facts in front of an agent, but it says nothing about which model that agent is running on. A frontier model handed a thin, well-curated payload and a small model handed a bloated one are the same misallocation pointed in opposite directions. Routing is the layer that makes that call, and it makes it per request, at production volume, mostly invisible when it's wrong.

The common failure mode is treating routing as a static assignment — "this workflow uses GPT-tier, that one uses the small model" — decided once at design time and never revisited. That's not routing, it's a hardcoded default with a routing-shaped name. Real routing is a decision under uncertainty, made per request, about a property you can't directly observe: how much reasoning this specific input actually requires.

Three Ways to Decide Where a Request Goes

Every routing mechanism reduces to a bet about task complexity, placed before the task is executed. They differ in what they bet on.

1. Rule-Based Routing: Betting on Surface Features

Regex triggers, keyword matches, prompt-length thresholds, an explicit task_type tag set by the caller. Cheap to build, cheap to run, and legible — anyone can read the rule and know why a request went where it went. The failure mode is equally legible in hindsight: surface features correlate with complexity poorly. A five-line prompt asking for a subtle security review and a five-hundred-line prompt pasting in boilerplate config for reformatting will hit a length threshold in the wrong order relative to what they actually need.

2. Classifier-Based Routing: Betting on a Learned Proxy

A small, fast model — or an embedding-similarity lookup against labeled examples — predicts a complexity or category label, and the router dispatches on that prediction. This corrects most of rule-based routing's surface-feature blindness, at the cost of a new failure mode: the classifier's training distribution becomes a hidden dependency. It is accurate exactly to the extent that production traffic resembles what it was calibrated against, and silently degrades as that resemblance erodes.

3. Cascade Routing: Betting Small, Then Verifying

Run the cheap model first. Check the output against a confidence signal — self-consistency across repeated samples, a lightweight verifier pass, a heuristic on output structure — and escalate to the expensive model only when that check fails. This is the only one of the three that pays for information about the actual task before committing the routing decision, rather than betting on a prediction made before execution starts. It costs an extra round trip on every escalated request, which is the correct price for not having to get the classifier right up front.

Architecture Rule: Route on measured task properties, not on static proxies for them. A confidence score from a cascade's cheap-model pass, an explicit complexity tag the caller is accountable for, or a task class's rolling escalation rate are properties of the actual request. Prompt length and keyword presence are not — they are correlated with complexity often enough to feel reliable and uncorrelated with it often enough to fail expensively.

The Cost of Getting It Wrong Runs in Both Directions

Misrouting up — sending a trivial task to a frontier model — is the failure mode everyone budgets for, because it shows up on the invoice. It's linear and it's visible: multiply the price difference by request volume and you have the exact cost of the mistake, sitting in a line item someone will eventually question.

Misrouting down is the one that doesn't show up until it's expensive somewhere else. A weak model handed a task past its actual capability doesn't fail loudly — it produces the same category of failure the Context Gap post named for stale context files: Confident Failure. The output is fluent, structurally correct, and wrong in a way that a downstream self-consistency check won't catch, because self-consistency measures whether a model agrees with itself, not whether it's right, and a model past its capability ceiling can agree with itself confidently and incorrectly every time.

The saved API cost from routing down is a real, immediate, and small number. The cost of the incident that Confident Failure eventually causes downstream is unmeasured until it happens and reliably larger than what was saved. Optimizing routing purely for cost-per-call is optimizing for the wrong side of that trade.

Coordination vs. Capability: The Multi-Agent Failure

The same misallocation shows up laterally, not just vertically, once you're running multiple agents against the same context instead of one agent against a tiered set of models. Role Bleed — two agents that both believe they own the same file, the same decision, the same slice of the task — gets diagnosed as a capability problem ("the model isn't smart enough to coordinate") when it's almost always a boundary problem: nobody defined, in a form either agent could check, who owns what.

Declaration solves this the same way it solves the Context Gap's version of the problem. An explicit ownership manifest — which directory, file, or decision belongs to which agent, checked at coordination time rather than inferred from convention — turns an implicit assumption into a fact either party can verify before acting. Two agents editing the same directory with no manifest are relying on both of them independently inferring the same boundary from context; two agents checking a manifest are relying on one file being correct. The second failure mode is easier to catch and easier to fix.

Routing Rules Decay Too

A routing rule calibrated once and left alone decays the same way a hand-maintained context file does, through a mechanism specific to routing: the models on either side of the rule keep changing out from under it.

Threshold drift. A complexity threshold tuned against last quarter's cheap model becomes miscalibrated the moment that model is upgraded. The new cheap model can now handle tasks the threshold still escalates — pure wasted spend, invisible unless someone is tracking escalation rate per task class over time rather than just aggregate cost.
Classifier staleness. A learned router trained on one traffic distribution degrades as the task mix shifts — new feature launches, new user segments, new failure modes nobody labeled examples for yet. Unlike a threshold, a classifier degrades silently in both directions at once: some tasks get over-escalated, others get under-escalated, and the aggregate cost metric can look flat while both error rates climb.
Manifest rot. Ownership manifests decay exactly like the paths-and-commands problem in context files — a directory gets renamed, split, or reassigned, and the manifest still names the old owner. The difference is the failure is worse than a broken link: it's two agents now confidently disagreeing about who's authoritative.

Making Misrouting Visible

None of the above is a reason to distrust routing infrastructure; it's a reason to instrument it the same way you'd instrument any decision made under a model you expect to drift.

Escalation rate per task class, tracked over time. A class whose escalation rate climbs is a threshold or classifier going stale in real time, not a one-off anomaly.
Shadow routing. Periodically run the tier the router didn't choose, in parallel, without serving the result — measure divergence between what was served and what the alternate tier would have produced, without paying the cost of serving both live.
Cost per completed outcome, not cost per call. Raw API spend rewards routing down. Cost per task that didn't require rework, escalation, or a human catching a Confident Failure downstream is the number that actually reflects whether the router is working.
Manifests as CI-checked artifacts, not documentation. The same path-assertion pattern that catches decayed context files applies here: a script that fails the build when a manifest names a directory, file, or agent that no longer exists.
Periodic reclassification audits. Spot-check a sample of routed requests against a human or stronger-model judgment of what tier they actually needed. This is the only mechanism on the list that catches drift a monitoring dashboard won't show, because it checks the router's decisions against ground truth instead of against its own history.

None of this replaces good context. A well-routed request still fails if the payload it's carrying is stale or incomplete — routing decides who gets to reason, the Context Gap post's declare/navigate/retrieve split decides what they get to reason with. But a router that can't tell a five-line security question from five hundred lines of boilerplate is spending its entire budget on the wrong axis, and neither problem is solvable by throwing a bigger model at it. The next failure mode sits underneath both: none of this routing logic means anything if the telemetry feeding it — request logs, error signals, the data a router or a coordinator has to make its decisions from — is itself inconsistent, unstructured, and unfit to reason over.