All articles
Engineering
8 min read
Inside the router: how we pick a model in 12ms
A look under the hood of the learned router that scores 40+ models on every single request.
Lena Park
Every request is a small auction
When a prompt enters the box, the router doesn’t ask which model is best in general. It asks which model is best for this exact request, under your latency and cost budget. Each candidate gets a predicted quality score from a lightweight model trained on millions of evaluated traces.
The cheapest model that clears your quality bar wins. If none do, the request escalates to a frontier model — and that decision is logged so you can audit it later.
Why 12 milliseconds matters
Routing overhead has to disappear into the noise of network latency. We compile the scoring model to run on CPU at the edge, cache embeddings for repeated prefixes and short-circuit obvious cases. The result: a median overhead of 12ms, and a p99 under 30.
Next article