Blog
Shadow mode
Part of the Unbiased glossary: the vocabulary of LLM pricing and infrastructure, defined with receipts instead of adjectives.
Shadow mode runs a candidate model on a copy of real production traffic without serving its outputs to users, so its quality and cost can be measured against the incumbent on identical inputs, at zero user risk. It is the standard technique for evaluating a model switch honestly.
Why it matters
Benchmarks are somebody else's workload. Shadow mode is yours: the same prompts, documents, and edge cases your system actually sees, scored side by side. It converts a model decision from a leaderboard argument into a measured comparison on the traffic that pays your bills.
The at-home version, measured
The lightweight variant is what we publish constantly: identical prompts to two models, both outputs and both metered bills kept. Our dashboard test billed $0.1462 on Fable 5 against $0.0196 on Pareto with both outputs working, embedded playable in the pricing breakdown so readers can judge quality themselves. Full shadow mode automates exactly that, continuously, on your traffic.
Common questions
Does shadow mode double my costs?
It adds the candidate's inference on mirrored traffic for the duration of the test, so budget the comparison window. Sampling (shadowing 5-10% of traffic) keeps the bill small while the comparison stays honest.
How long should a shadow test run?
Long enough to cover your traffic's real variety, typically one to two representative weeks. A day of easy traffic proves nothing about the tail.
What do I measure?
Task success on your own criteria, cost per completed task, and latency, on identical inputs. If you only measure vibes, shadow mode is theater.