Same name
same list price, measurably different quality across providers of one model
Blog
Provider variance: the catalog's quiet problem
The same open-weights model served by two providers is not the same model in practice. OpenRouter's own telemetry proved it; here is what that means for anyone routing through a catalog.
Billions
of tool calls in the telemetry that pushed OpenRouter to build Exacto
1 defense
a fixed eval set run on schedule, whoever your routing layer is
Provider variance means the same open-weights model served by different hosts returns measurably different quality: different quantization, serving stacks, and configs produce different outputs at the same list price. OpenRouter measured tool-call accuracy varying by provider across billions of calls and launched Exacto in response. Defend with scheduled evals, not provider labels.
What the variance is
An open-weights model on a catalog is not one thing; it is one set of weights served by many independent hosts, each choosing quantization, serving stack, context handling, and configuration. Those choices change outputs. The price column says the providers are interchangeable; the outputs say otherwise.
This is specific to the open-weights side of catalogs. First-party APIs (Anthropic serving Claude, OpenAI serving GPT) do not have a multi-host variance problem; what they serve is what you get.
The vendor's own receipts
Credit where due: OpenRouter measured this rather than deny it. Telemetry across billions of tool calls showed tool-call accuracy varying by provider for the same model, and Exacto launched specifically because of it. That is the right response and also an admission worth reading twice: the catalog's operator confirmed that 'same model, same price' does not mean 'same output.'
For a buyer, the conclusion is not 'avoid catalogs.' It is that provider selection is a second selection job stacked on top of model selection, and it is quality-bearing.
How to defend your quality bar
The defense is the same whoever routes for you: a fixed eval set, run on schedule, against the exact route your production traffic takes. Latency SLOs without a quality canary miss the failure mode that actually hurts, the silent regression; we keep a glossary entry on it because it is the category's most expensive surprise.
This is also the argument for receipts at whatever layer picks the model. We publish ours, benchmark-by-benchmark with the harness public, on the model card, and the honest generalization is: any layer that selects models for you should be able to show you measured evidence of what its selections deliver. If it cannot, your eval set is the only receipt you will ever have.
Questions, answered
Why would the same model differ by provider?
Different hosts make different serving choices: quantization level, inference stack, context and sampling handling. Same weights, different pipeline, different outputs.
Does this affect first-party APIs?
No; variance across hosts is specific to open-weights models served by multiple providers. First-party models have one host: the lab itself.
What is Exacto?
OpenRouter's response to its own telemetry: after measuring tool-call accuracy varying by provider across billions of calls, it launched Exacto to steer tool-calling traffic toward better-performing providers.
How do I detect a silent quality regression?
A fixed eval set run on schedule against your production route, with results tracked over time. Spot checks miss drift; scheduled canaries catch it.
Compare any two models
VS
Rates verified 2026-07-28. "Measured task" = our identical dashboard-generation prompt, metered where marked ✓ and list-math otherwise. Verbosity from the Verbosity Index, Edition 1. Data: prices.json.
Run the comparison on your own traffic. $100 to verify.
Buy Pareto API credits