Shadow mode runs a candidate model on a copy of real production traffic without serving its outputs to users, so its quality and cost can be measured against the incumbent on identical inputs, at zero user risk. It is the standard technique for evaluating a model switch honestly.
Why it matters
Benchmarks are somebody else's workload. Shadow mode is yours: the same prompts, documents, and edge cases your system actually sees, scored side by side. It converts a model decision from a leaderboard argument into a measured comparison on the traffic that pays your bills.
The at-home version, measured
Preliminary Pareto 26.10 scores and cost per task on four benchmarks are on the model card. These results may change before final publication, and no composite score is published.
Before you mirror live traffic, run your own prompts and graded test cases against Pareto next to your current model: the eval recipe takes one prompt file and one CSV.
How to run a shadow test
- Write the decision rule first. Decide what the candidate must match or beat (task success, cost per completed task, latency) before any traffic moves, so the result cannot be argued after the fact.
- Sample live requests. Copy a random share of production requests to the candidate. 5-10% is usually enough, and it keeps the extra bill small.
- Serve only the incumbent. Users get the current model's answer. The candidate's answer is logged next to it under the same request ID and never shown.
- Grade both answers the same way. Use your own pass criteria: correct result, valid format, no refusal. Human graders or a fixed rubric both work, as long as neither knows which model wrote which answer.
- Compare by request type. Totals hide the tail. Break results out by the kinds of requests you see, because a switch usually fails on the rare, hard ones.
Shadow mode, A/B test or offline eval?
They answer different questions, and most teams use all three in order.
| Method | Inputs | Users see the candidate? | What it tells you |
|---|---|---|---|
| Offline eval | Cases you wrote | No | Does it pass my known cases? |
| Shadow mode | Sampled live traffic | No | Quality, cost, latency on real traffic |
| A/B test | Live traffic | Yes, the test group | Do users do better with it? |
What a shadow test costs
Mirrored requests are billed at the candidate's normal token rates; the incumbent's bill does not change. With Pareto at $0.80 per million input tokens and $3.20 per million output tokens (pricing), a week that mirrors 10 million input tokens and 2 million output tokens costs $8.00 plus $6.40: $14.40 for the whole comparison. Shadowing sends live traffic to Pareto, so confirm production use in a separate written agreement with Unbiased before you start.
Common questions
Does shadow mode double my costs?
It adds the candidate's inference on mirrored traffic for the duration of the test, so budget the comparison window. Sampling (shadowing 5-10% of traffic) keeps the bill small while the comparison stays honest.
How long should a shadow test run?
Long enough to cover your traffic's real variety, typically one to two representative weeks. A day of easy traffic proves nothing about the tail.
What do I measure?
Task success on your own criteria, cost per completed task, and latency, on identical inputs. If you only measure vibes, shadow mode is theater.
Is shadow mode the same as an A/B test?
No. In an A/B test some users get the candidate's answers. In shadow mode nobody does: the candidate runs on copies of live requests and its answers are only logged and graded. Shadow first, then A/B test if you need to measure what users do with the new answers.
Compare any two models
List rates and dated competitor measurements: prices and measured bills. Preliminary Pareto 26.10 benchmark costs per task from October 1, 2026 are on the model card. Verbosity: Edition 2.
Try it on one part of your codebase
Set Pareto up in the coding agent you already use and give it a week of real work. Pareto starts at $10 a month.
Read https://unbiased.ai/glossary/shadow-mode/ and tell me what matters for my setup. If Pareto fits, read https://unbiased.ai/works-with/agent.md, find the coding agent I use on this machine, and set it up to use Pareto. Ask before changing any file, and never ask me to paste my API key into chat.
It asks before changing files and never needs your key in chat. Read the agent runbook
