Blog
Silent quality regression
Part of the Unbiased glossary: the vocabulary of LLM pricing and infrastructure, defined with receipts instead of adjectives.
A silent quality regression is a drop in model output quality that arrives without an announcement, version change, or price signal, typically when a provider changes serving infrastructure, quantization, or backend routing. The API responds normally; the answers are quietly worse.
Why it matters
Engineering notices latency and finance notices price, but nobody's dashboard watches answer quality, so regressions surface as a vague sense that the model got dumber, weeks late, after the damage. In production pipelines the cost is failed tasks and silent rework, which never appear on the invoice.
The evidence this is real, from the vendors themselves
OpenRouter measured tool-call accuracy across billions of calls and found the same model served by different providers returns measurably different quality, launching its Exacto tier specifically in response. That is an aggregator documenting variance inside its own catalog. The defense is continuous measurement at the layer that picks the model: same-harness comparisons with published receipts, which is how we publish ours.
Common questions
How do I detect a silent regression?
A fixed evaluation set run on a schedule against production endpoints, with results tracked over time. Anything less and you are relying on user complaints as your monitoring.
Why would a provider degrade quality silently?
Capacity economics: quantization and serving optimizations cut cost per request, and the incentive to announce a quality trade is weak. Assume drift; verify instead of trusting.
Does this affect first-party APIs or just aggregators?
Both have changed serving behavior without fanfare historically; aggregators add the extra variable of which provider serves you at all. Measurement is the answer in either case.