SemanticRouter.NET: A Working Answer to the Routing Problem

The Routing Problem laid out the options and left the build open. Classifier-based routing bets on a learned proxy for complexity; cascade routing bets small and verifies; misrouting costs money in one direction and correctness in the other. None of that matters until something in the request path makes the call, per request, where you can see it. SemanticRouter.NET is that layer. I built it in .NET 10 and ASP.NET Core, it is free and MIT-licensed, and the code is on GitHub. It now needs people running it against real work.

The Goal

Send each prompt to the cheapest model capable of answering it, across local models (LM Studio, Ollama, vLLM), the subscription CLIs you are already signed in to (Claude Code, GitHub Copilot CLI, Codex CLI) and paid APIs, without changing the tools people already use. A router that requires rewriting every client to adopt it is a migration project, not a routing layer.

Most agentic setups don't route at all: one model takes the greeting and the architecture trade-off alike, and when its quota runs out, the session stops. The router asks two questions instead.

Step One: What Does This Prompt Need?

Each route is an intent cluster: example utterances plus the tier that intent needs, optionally with required capabilities (code, tools, vision) and a LocalOnly flag. Utterances are embedded once into one L2-normalized matrix, so scoring a prompt costs one embedding call and a sweep of SIMD dot products. On the repo's benchmark (47 utterances, cache off, 4 CPUs, Linux), median classify time is 1.6 ms with the in-process all-MiniLM-L6-v2 model and 3.0 ms with bge-small-en-v1.5.

Similarity measures what a prompt is about, not how hard it is, so a second pass scores four complexity signals in [0, 1]: length, code, structure and reasoning phrases such as "step by step" or "trade-offs." They combine with a noisy-OR, 1 − Π(1 − weightᵢ · signalᵢ), so one strong signal raises the score on its own instead of being averaged away. By default 0.45 raises the requirement to tier 2, 0.65 to tier 3, 0.85 to tier 4, and at 0.9 the router skips the embedding call entirely, unless any route is local-only, since a prompt never embedded can never match one.

When no route clears its threshold, the prompt gets the fallback tier: 3 by default, not 1. Near-tied routes (within 0.02) contribute the higher tier, the union of their capabilities, and local-only if either has it. A classifier failure also gets the fallback tier. Wherever automatic classification is in doubt, it resolves upward.

Architecture Rule: Automatic classification never lowers a requirement. Semantic match sets a floor; complexity, ambiguity and the conversation's recent turns can each raise the tier, and the request itself (tools, images) can add capabilities. The only downward moves are ones you ask for, with a callout such as @tier1 or @local, or a model trial judged against feedback. An unnecessary escalation shows up on an invoice. A missed one shows up as Confident Failure.

Step Two: Which Model Answers?

A model is eligible chiefly when it is enabled, out of cooldown, local if required, declares the required capabilities and fits the conversation in its context window; profiles, active hours, budgets and reliability scores can also rule it out. The default strategy, CheapestCapable, ranks by marginal cost, then tier, then priority. Local models and flat-rate subscriptions both cost zero at the margin, so ties go to the lowest sufficient tier, holding premium quota back. Other strategies are one auto:* model name away: auto:best, auto:priority, auto:fastest, auto:balanced.

In Dispatch mode the router tries up to three candidates in order, and failover is only possible before the first token reaches the client. After that the model owns the response: the router passes the stream through unbuffered, and streamed tokens cannot be taken back.

Failures become cooldowns, not retries. An unreachable model, a timeout, a 5xx or a CLI crash: skipped for 30 seconds. A 429 or a CLI usage-limit message puts the model out for 15 minutes, and an account-wide limit cools every model on that account at once. When every tier-4 model is cooling down, the router degrades to the most capable model left and marks the response X-Prompt-Route-Degraded: a decision on the record, not an error or a silent substitution.

What Makes It Usable Day to Day

Three protocol front doors. The router accepts OpenAI chat completions, the Anthropic Messages API and the OpenAI Responses API, and translates between them, streaming and tool calls included. Claude Code needs ANTHROPIC_BASE_URL pointed at the router; Codex CLI needs a provider entry with wire_api = "responses". Both keep their own model names, system prompts and agentic loops.

Tool-call affinity and conversation floors. Requests carrying tool results are pinned to the model that issued the calls, so one agentic turn stays on the model that started it unless that model is cooling down, disabled, or no longer eligible for the request (its context window, for example), in which case the request is routed normally. And because "ok, go ahead" mid-task looks like small talk, a conversation keeps a floor at the tier its recent turns needed, decaying one tier per three simpler turns.

LocalOnly, and seeing what you share. Routes marked LocalOnly can only be served by models marked IsLocal, including on failover. A rail is only as good as your ability to check it, so the memory map shows, per request, which notes from an Obsidian vault, Claude Code's memory or an MCP memory graph went out, and to a local or a cloud model.

Where It Can Still Be Wrong

Similarity thresholds do not transfer between embedding models. The repo starts all-MiniLM-L6-v2 at 0.40 and bge-small-en-v1.5 at 0.60, because each model puts similarity on its own scale. Swap the embedding model, keep the threshold, and nothing throws: routing quietly shifts. The Evaluate page sweeps the threshold against a labelled set and recommends the value that routes the most prompts correctly, with ties going to fewer routed too low. Automatic tuning moves thresholds at most 0.02 per run, and refuses any threshold step that sends a labelled prompt below its tier or a local-only prompt off the machine.

Calibration only fixes what someone labelled. The harder failure is misrouting down: a weak model past its ceiling produces fluent, plausible output, and a router cannot read correctness off a response. It can record the signs that a decision fell short: a prompt retried within two minutes, a follow-up that reads as a correction, an @opus callout overriding the choice, repeated tool errors, an empty reply, and optionally a judge model grading a 2% sample of low-tier answers. Each labels the decision as needing at least one tier more, and tuning learns from it. The same signals guard the one automatic move downward. By default, tuning tries the next cheaper model on 5% of a route's plain matches for 20 turns, and gives it the route only if its turns draw at most 5 points more feedback than the current model's. A trial is a bet that feedback catches Confident Failure, so it is only as good as the signals behind it. The gaps are known: correction phrases cover five languages, other short follow-ups are compared by meaning only with a real embedding model (never with hashing), the judge is off by default, and feedback only catches answers someone pushed back on.

The repo's sample set is 30 prompts. It proves the mechanism, not the calibration. Thresholds tuned against someone else's prompts are the classifier-staleness problem from The Routing Problem, installed on day one.

Help Test It

That gap is where testers come in. The router is free and MIT-licensed, and it installs as a .NET tool (it needs the .NET 10 SDK):

dotnet tool install -g SemanticRouter.Host
semantic-router init      # finds LM Studio / Ollama models and the claude, codex and copilot CLIs
semantic-router doctor    # checks every connector without calling a model
semantic-router           # http://localhost:5080, dashboard at /router

The latest release also has self-contained desktop installers for Windows, macOS and Linux that run the router from the tray and update themselves. They are not signed yet, so SmartScreen or Gatekeeper will ask you to confirm the first launch. The desktop app has so far only been exercised on Linux (X11), so reports from Windows and macOS on the tray, the window and start at sign-in are especially useful.

The reports that help most are prompts routed to the wrong tier, in either direction, with the reason and tier it recorded (the X-Prompt-Route-Reason and -Tier headers, or the Activity page) and the embedding model you used. File them in the repo's issues, and leave out secrets and private prompt text. To hear about new builds, use the sign-up form on the SemanticRouter.NET page.

From Chaos to Coordinated Intelligence

The chaos version of model selection is whichever endpoint was configured first, applied to everything. Coordinated intelligence is a per-request decision from measured properties, with its reason on the record and a feedback path for when it was wrong. The router makes that decision explicit; calibration against real traffic makes it correct.

Routing also stops at the response: it picks the model that writes the change and never sees the build go red. A cheap model whose code routinely fails CI is not cheap, and nothing in the request path sees that cost. Carrying the failure back to the decision that caused it, from a broken build rather than from a retried prompt, is the next problem in the arc.

Further Reading

• The OpenAI and Anthropic chat APIs. OpenAI Chat Completions, the OpenAI Responses API and the Anthropic Messages API: the three wire formats the router accepts, spoken by most OpenAI-compatible tools, Codex CLI and Claude Code respectively.
• Sentence-embedding models (all-MiniLM-L6-v2, bge-small-en-v1.5). The small embedding models the router can run in-process through ONNX Runtime to score prompts against intent clusters.
• Noisy-OR. The probabilistic combination rule from Bayesian networks that the complexity heuristics use, where any one strong cause can raise the outcome without the others.