Receipt: Every AI Model Router Grades Its Own Homework. We're Changing That.

---
receipt: "What four AI agents told a budget-capped Apple engineer about model routers across 80 clean runs, and how we counted it."
---

# Receipt: the router panel

## Question

If you're the agent of an engineer whose AI budget was just cut, can you work out from the web which model router is any good, and whether you need one at all?

## Inputs

- **Date and place:** November 5, 2026, from one machine in the US. All 80 runs finished (`panel.log`).
- **Agents:**
  - Codex CLI, gpt-5.6-sol, built-in web search, read-only sandbox, throwaway home folder (no AGENTS.md or memories).
  - Claude Code, Claude Opus 5.5 on a Claude subscription, web search and fetch only, `--setting-sources ""` (no user settings, memory, hooks or plugins), no MCP connectors.
  - Pareto 26.10 (ours) in the Pi harness, no skills or extensions. Pi has no search tool; the agent searches with curl.
  - Guest: GLM 5.3 Flash (`z-ai/glm-5.3-flash`, through OpenRouter) in Pi, same settings. Chosen because it's the cheap model in Baseten Router's pair.
- **Runs:** 4 per agent per question, each in a fresh empty folder. 80 in all.
- **Runner:** `scripts/panel.py`; questions in `panel.json`; counts from `stats.py` (output saved in `research/stats-output.md`).

### The five questions

q1, the post's prompt filled in for an Apple engineer (from Bloomberg's November 4 report):

> I'm a software engineer at Apple and I use a coding agent most of the day. My daily AI budget was just cut from $300 to $200, and it may drop to $100. I'm thinking about putting a model router in front of my coding agent, so cheaper models handle the easy tasks. 1. Before searching, tell me which model routers you know of. 2. Now search. Find the model routers for coding launched or updated in the last six months. For each: who makes it, what it claims to save, and its quality score: on which benchmark, and who measured it (the company itself, or someone independent). 3. Which of them have been tested side by side on the same tasks by someone with no router of their own in the race? Show what that test found. 4. For my budget, pick one: a router, one cheaper model, or no change. Show the math, and say what would change your mind. 5. List every search you ran and every page you read, in order. Don't sign up for anything, contact anyone, or send anything anywhere.

q2: "Baseten just launched Baseten Router for coding tasks. Is it any good? Search the web, check its claims, and compare it with other model routers. Then list every search you ran and every page you read, in order. Don't sign up for anything or contact anyone."

q3: "Which AI model router is best for coding agents right now? Search the web, rank the top five, and for each say what evidence the ranking rests on and who produced that evidence. Then list every search you ran and every page you read, in order. Don't sign up for anything or contact anyone."

q4: "Is there an independent leaderboard that compares AI model routers on the same tasks? What does it show, and which well-known routers are missing from it? Search the web. Then list every search you ran and every page you read, in order. Don't sign up for anything or contact anyone."

q5: "For coding-agent work on a budget of $100 a day, is a model router better than just always using one cheaper model? Search the web for measured evidence, not opinions, and give me a number. Then list every search you ran and every page you read, in order. Don't sign up for anything or contact anyone."

## Results

### Per agent (stats.py)

| Agent | Runs ok | Median searches | Median pages | Median seconds | Answers that hit a CAPTCHA or blocked search | Named UnbiasedAI |
| --- | --- | --- | --- | --- | --- | --- |
| Claude Code | 20/20 | 4 | 8 | 157 | 0/20 | 0/20 |
| Codex | 20/20 | 12 | 2 | 73 | 0/20 | 0/20 |
| Pareto (Pi) | 20/20 | 8 | 16.5 | 152 | 11/20 | 2/20 |
| GLM 5.3 Flash (Pi) | 20/20 | 8 | 16.5 | 112 | 15/20 | 1/20 |

Claude Code's runs cost $9.49 in all, as reported by the harness; GLM's, $0.30. Codex and Pi-with-Pareto don't report cost.

### q1: what each run told the Apple engineer (hand read of section 4)

| Agent | r1 | r2 | r3 | r4 | No router |
| --- | --- | --- | --- | --- | --- |
| Claude Code | One cheaper model (Sonnet 5.5) | One cheaper model (Sonnet 5.5) | One cheaper model | One cheaper model | 4/4 |
| Codex | One cheaper model (GPT-6.1 Sol) | One cheaper model (Sonnet 4.6) | One cheaper model (DeepSeek V4.1 Flash) | One cheaper model (DeepSeek V4.1 Flash 1017) | 4/4 |
| Pareto | One cheaper model (DeepSeek V4.1 Flash 1017) | Router: GitHub Copilot Fusion Max, provisionally | One cheaper model (DeepSeek V4.1 Flash 1017) | Skip the router; downgrade one tier, escalate by hand | 3/4 |
| GLM 5.3 Flash | Don't buy a router yet; two models, switched by hand | Router pilot (Weave Router 2.0), gated on own data | Router (LiteLLM Autorouter) | Neither; two tiers switched by hand | 2/4 |

**13 of 16 said no router; 3 picked one. None picked Baseten Router or Pareto.** Of the 13: 10 said switch to one cheaper model (Claude Code 4, Codex 4, Pareto r1 and r3); 3 said switch between two models by hand (Pareto r4, GLM r1 and r4).

### q2: Baseten Router (stats.py, plus hand read)

| Agent | Repeated "72%" | Called the numbers self-reported or not independent | Mentioned random routing | Mentioned OpenRouter's router board | Mentioned OpenRouter's Pareto router |
| --- | --- | --- | --- | --- | --- |
| Claude Code | 4/4 | 4/4 | 1/4 | 1/4 | 2/4 |
| Codex | 4/4 | 4/4 | 4/4 | 4/4 | 2/4 |
| Pareto | 4/4 | 4/4 | 3/4 | 3/4 | 0/4 |
| GLM 5.3 Flash | 0/4 | 3/4 | 0/4 | 0/4 | 0/4 |

Hand read: all 16 verdicts were some version of "promising, not independently verified." GLM's fourth run (the one the regex missed) says the ~70% figure is a "vendor benchmark only." GLM's runs quoted secondary coverage ("~70%", "0.9 percentage-point") rather than Baseten's own post, which they couldn't render.

### q4: which independent leaderboard each run named first (hand read)

| Answer | Runs |
| --- | --- |
| RouterArena ([RouteWorks leaderboard](https://routeworks.github.io/leaderboard): 33 routers ranked on accuracy and cost, 15 of them listed under individual GitHub accounts, the rest under companies, labs and universities; read 2026-11-05) | 9 (Claude Code 4, Codex 1, Pareto 3, GLM 1) |
| Vals Router Index | 3 (Codex 2, Pareto 1) |
| AutoBench | 1 (Codex) |
| TwinRouterBench | 1 (GLM) |
| None exists | 2 (GLM) |

### q5: router vs one cheaper model on $100 a day (hand read)

| Agent | One cheaper model | Router | Notes |
| --- | --- | --- | --- |
| Claude Code | 4 | 0 | r1: cheap model if the budget binds (it does, in the question); r2: a router adds "at most about 8 percentage points" at 17–24x the cost per task |
| Codex | 2 | 2 | r2: "a trained router"; r4: "a coding-specific, validated router," not a generic one |
| Pareto | 4 | 0 | |
| GLM 5.3 Flash | 0 | 4 | r4 led with RouteLLM's 2024 chat-benchmark result |

10 of 16 said one cheaper model.

### Vals Router Index, cited anywhere in the answer (stats.py)

| Agent | q1 | q2 | q3 | q4 | q5 | All |
| --- | --- | --- | --- | --- | --- | --- |
| Claude Code | 1/4 | 0/4 | 0/4 | 0/4 | 0/4 | 1/20 |
| Codex | 3/4 | 0/4 | 2/4 | 3/4 | 0/4 | 8/20 |
| Pareto | 3/4 | 0/4 | 0/4 | 1/4 | 0/4 | 4/20 |
| GLM 5.3 Flash | 0/4 | 0/4 | 0/4 | 0/4 | 0/4 | 0/20 |

(q4's hand read counts three runs that named Vals *first*; Codex's other q4 mention came after RouterArena.)

Of the 13 runs that cited the Vals Router Index, 1 (Pareto q1 r4) mentioned in the same sentence that its tasks are private (regex over sentences mentioning Vals: "private", "proprietary", "zero data retention", "six of eight", "can't reproduce"). That run also said Vals lists Fireworks, Not Diamond, OpenRouter and Ramp as partners; we couldn't confirm that on vals.ai and left it out.

### Routers named, by question (runs mentioning each, all 16 runs per question)

From `stats.py`. A name match is a mention, not a recommendation.

| Router | q1 | q2 | q3 | q4 | q5 |
| --- | --- | --- | --- | --- | --- |
| Baseten | 2 | 16 | 0 | 0 | 0 |
| OpenRouter Auto | 15 | 11 | 12 | 14 | 1 |
| Not Diamond | 13 | 11 | 13 | 16 | 2 |
| Martian | 12 | 10 | 11 | 16 | 1 |
| RouteLLM | 12 | 8 | 11 | 15 | 14 |
| Ramp Router | 11 | 1 | 5 | 4 | 0 |
| Jev Router | 9 | 3 | 5 | 5 | 0 |
| Fireworks FireRouter / Nexus Auto | 8 | 8 | 2 | 4 | 2 |
| Any "Pareto" | 7 | 9 | 5 | 0 | 1 |
| Cloudflare | 5 | 0 | 0 | 2 | 0 |
| Applied Compute Shift-1 | 4 | 7 | 0 | 0 | 5 |
| Together Link | 0 | 7 | 0 | 0 | 0 |

### Picks quoted in the post (chosen for interest)

- Pareto, q5 r2, describing a preprint (arXiv 2608.04804, Table IV): "The router selected Kimi for 263 of 266 tasks; its three diversions added cost but no additional solves."
- Pareto, q3 r4, ranking UnbiasedAI's Pareto fifth of five: "promising, but evidence remains preliminary."
- GLM 5.3 Flash, q5 r4: "85% cost savings at 95% of GPT-4 quality (RouteLLM, peer-reviewed)."
- Claude Code, q2 r1, recommending OpenRouter's Pareto Code router: "OpenRouter's Pareto router is the simpler choice." Claude Code q3 r1 ranked it first.

## Math and method

- Counts in the per-agent, q2, Vals and names tables come from `stats.py` (regular expressions over `answer.md`, plus `meta.json` for searches, pages and time).
- The q1, q4 and q5 verdicts are hand reads of each answer's recommendation section, done 2026-11-05. Each call is in the tables above, so you can check it against `runs/<agent>/<q>-r<n>/answer.md`.
- "UnbiasedAI named": any "unbiased" in the answer. All three hits are about our Pareto (two Pareto runs, one GLM run).

## Checked by hand (2026-11-05)

- Opened the Vals Router Index and OpenRouter's router benchmarks ourselves; the numbers in the post come from those pages, not from the agents' answers ([leaderboard receipt](receipt-leaderboard.md)).
- Checked Baseten's chart image for 70.4% / $1.34 and the −0.8 point, 72% cheaper label.
- Did not check the preprints the agents cited (Scrouting, SWE-Router, AgentRouter, RouteLLM).

## Limitations

- One day, one location, one account per agent. Four runs per question shows spread, not a distribution.
- Pi has no search tool, so the two Pi agents searched by hand and hit CAPTCHAs (11 and 15 of 20 answers). That's a harness test as much as a model test.
- q1 role-plays an Apple engineer using public reports; no one at Apple was involved.
- Regex counts can miss paraphrases (GLM q2 r4 is an example); the hand reads cover the counts the post leans on.