Receipt: Every AI Model Router Grades Its Own Homework. We're Changing That.

---
receipt: "Who grades the routers: OpenRouter's router benchmarks, Vals's Router Index, Arena and Artificial Analysis. Who is on each, what each shows, and the catch with each"
date: 2026-11-05
---

# Receipt: the router scoreboards

## Question

Of the model routers launched or relaunched this year, which ones have been graded side by side on the same tasks, and what does that grading show? And for a coding agent on a $100 or $200 daily budget, what does each option buy?

## Part A: Vals Router Index (no router of its own; private tasks)

Source: [vals.ai/benchmarks/vals_router_index](https://www.vals.ai/benchmarks/vals_router_index), "Updated 11/3/2026", rendered in a headless browser 2026-11-05; text saved in `research/vals-router-index-2026-11-05.txt`. Confidence: reported (Vals's results; we read the page, we didn't rerun anything). Vals describes itself as an independent AI evaluator; we found no router product of its own.

Method, per Vals: six router setups run "through the same agents, tasks and graders as every model on the Vals Index," about 1,250 tasks from eight benchmarks of agent work in finance, coding, law and tax; coding reported separately (Terminal-Bench, Vibe Code Bench, Code Migration). Each router "with its default configuration." Six of eight benchmarks are private, run under zero-data-retention agreements.

Headline, quoted: "None of the six routers sits on the cost and accuracy frontier: each was beaten on both score and cost by at least one single model."

### Full index

| Rank | Router | Accuracy | Cost / test | Best single model at or below its cost |
| --- | --- | --- | --- | --- |
| 1 | Nexus Auto with Sol (Fireworks) | 60.02% | $1.25 | DeepSeek V4.1 Flash 1017, 61.3%, $0.66 |
| 2 | Router (Ramp) | 57.80% | $0.96 | DeepSeek V4.1 Flash 1017, 61.3%, $0.66 |
| 3 | Nexus Auto with Opus (Fireworks) | 56.46% | $9.22 | Wonder Full, 66.8%, $6.81 |
| 4 | Jev Router (OpenRouter) | 55.65% | $5.30 | GPT-6.1 Sol, 63.9%, $3.61 |
| 5 | Auto Router (OpenRouter) | 52.20% | $0.82 | DeepSeek V4.1 Flash 1017, 61.3%, $0.66 |
| 6 | Fugu Max (Sakana) | 48.11% | $3.91 | (pool undisclosed; "ten single models beat it on both score and cost") |

### Coding only

| Router | Score | Cost / test | Best in its pool | Best single model at or below its cost |
| --- | --- | --- | --- | --- |
| Nexus Auto with Sol (Fireworks) | 61.5% | $2.48 | GPT-6.1 Sol, 73.0%, $5.97 | DeepSeek V4.1 Flash 1017, 62.7%, $0.90 |
| Router (Ramp) | 57.1% | $1.82 | GPT-6.1 Sol, 73.0%, $5.97 | DeepSeek V4.1 Flash 1017, 62.7%, $0.90 |
| Jev Router (OpenRouter) | 54.0% | $11.14 | Claude Sonnet 5.5, 80.8%, $25.25 | GPT-6.1 Sol, 73.0%, $5.97 |
| Nexus Auto with Opus (Fireworks) | 48.8% | $20.42 | Claude Opus 5.5, 63.0%, $39.40 | GPT-6.1 Sol, 73.0%, $5.97 |
| Auto Router (OpenRouter) | 47.6% | $1.87 | DeepSeek V4.1 Flash 1017, 62.7%, $0.90 | DeepSeek V4.1 Flash 1017, 62.7%, $0.90 |

### A $200 day on Vals's coding tests (estimate)

Tasks per day = 200 ÷ cost per test. Claude Opus 5.5: 200 ÷ 39.40 = 5.1. DeepSeek V4.1 Flash 1017: 200 ÷ 0.90 = 222.2. Ratio: 39.40 ÷ 0.90 = 43.8, "about 1/44th." Score gap: 63.0 − 62.7 = 0.3 points. Best router, Nexus Auto with Sol: 200 ÷ 2.48 = 80.6. Ramp's Router: 200 ÷ 1.82 = 109.9.

### Not on Vals (checked by name, 2026-11-05)

Baseten Router, Applied Compute Shift-1, Together Link, Cloudflare Auto Router, Cursor Router, and UnbiasedAI's Pareto. Vals: "We're eager to evaluate other routers."

### The catches (2026-11-05)

- Private tasks. The page: "Six of its eight benchmarks are private, and we run them under zero-data-retention agreements, so model providers cannot keep or train on the tasks." Nobody outside Vals can rerun them.
- Mostly not coding. Eight benchmarks in finance, coding, law and tax; three are coding (Terminal-Bench, Vibe Code Bench, Code Migration), reported separately.
- One setting per router. "Each router ran with its default configuration." OpenRouter's Auto Router "ran at its cost-saving default."
- Moving numbers. The launch note on vals.ai's front page (read 2026-11-05) says the index "runs five model routers," lists "FireRouter (Fireworks)," and says "Router (Ramp) came closest, at 57.8% for $0.96 per test." The live index page lists six setups, with Nexus Auto with Sol (Fireworks) first at 60.02%. And the page prints its Results paragraph twice with different figures (see "Page quirk").
- In our panel, 1 of the 13 runs that cited Vals mentioned that its tasks are private ([panel receipt](receipt-panel.md)).

### Page quirk

The page prints its "Results" paragraph twice with different figures (DeepSeek beating Nexus Auto with Sol "for about half the cost" vs "for about a third of the cost"; beating Ramp's Router by "about 3.5 points" vs "about 5.5 points"). They look like the full-index and coding versions of the same paragraph. We used the tables, not those sentences.

## Part B: OpenRouter's router benchmarks (shared, not neutral)

### Inputs

| Input | Value | Source | Confidence |
| --- | --- | --- | --- |
| OpenRouter, Model routing benchmarks | Routers graded: Auto (now powered by Jev Router), Auto Code, Fugu Max, Fugu Ultra v2, Pareto, plus NVIDIA Switchyard model switching (7 model pairs) and single-model baselines. 151 published router and baseline runs. "Last benchmark run Nov 4, 2026, 4:38 PM UTC." | [openrouter.ai/benchmarks/routers](https://openrouter.ai/benchmarks/routers), rendered in a headless browser 2026-11-05; text saved in `research/openrouter-routers-2026-11-05.txt` | Verified (we read the page) |
| Benchmarks on the board | Deep SWE, SWE Atlas QA, Terminal Bench 4.0, Tau3 Banking, MMMU Pro Vision | same | Verified |
| Index weighting | Model Routing Index: 60% quality, 20% speed, 20% cost, 0–10 relative to the best result on each benchmark | same | Verified |
| Apple daily caps | Engineers cut from $300 to $200 a day in October; some teams may go to $100; many non-engineers already at $100, some may go to $25 | Bloomberg, 2026-11-04, via [9to5Mac](https://9to5mac.com/2026/11/04/report-apple-reduces-employees-ai-token-allowances-as-ternus-moves-to-cut-costs/) and [MacRumors](https://www.macrumors.com/2026/11/04/apple-reducing-employee-ai-spending/) | Reported (we couldn't open Bloomberg's article) |

### Results

#### 1. Who's on the board

On it: OpenRouter's own Auto and Auto Code, Fugu Max and Fugu Ultra v2, Pareto (ours), and NVIDIA's Switchyard.

Not on it (checked by name against the page text, 2026-11-05): Baseten Router, Fireworks FireRouter, Together Link, Applied Compute Shift-1, Ramp Router, Cloudflare Auto Router, Cursor Router, GitHub Copilot Fusion.

#### 2. Deep SWE (the long-horizon coding test; the closest thing on the board to Baseten's DeepSWE 1.1)

Rows as the page lists them, hand-copied. Cost per success is the page's own column.

| Row | Pass rate | Cost / task | Cost / success | Time / task |
| --- | --- | --- | --- | --- |
| Kimi K3.1 Pro (single model) | 77.9% | $0.85 | $1.09 | 18.6m |
| Pareto (router, ours) | 74.3% | $0.19 | $0.26 | 7.5m |
| Auto Code (router, OpenRouter) | 74.3% | $1.98 | $2.66 | 15.2m |
| Switchyard · GLM 5.3 Flash + Claude Opus 5.5 (high) | 73.5% | $2.47 | $3.36 | 17.4m |
| Claude Opus 5.5 (row marked "Failed" on the page) | 70.5% | $7.76 | $11.00 | 51.9m |
| Auto, Medium tier (router, OpenRouter, Jev-powered) | 69.0% | $0.57 | $0.83 | 9.6m |
| Claude Opus 5.5 | 67.7% | $1.54 | $2.27 | 7.0m |
| Claude Opus 5.5 | 61.1% | $0.52 | $0.85 | 2.6m |
| Fugu Max (router) | 33.9% | $3.00 | $8.85 | 25.4m |
| GLM 5.3 Flash (high) | 31.0% | $0.34 | $1.10 | 27.0m |

The page lists Opus 5.5 three times on Deep SWE; the text dump doesn't say which effort setting each row is. The index matrix shows Opus 5.5 (high) as "Unknown" on Deep SWE, and low and medium at 8.4 each.

#### 3. Where Pareto loses (same size as the wins)

| Benchmark | Pareto | Best on that benchmark |
| --- | --- | --- |
| Tau3 Banking | 35.1% at $0.15/task | Kimi K3.1 Pro 54.6%; best router Auto (Medium) 53.1% at $0.60 |
| MMMU Pro Vision | 75.5% (18th of 22 rows) | Gemini 4 Argon 87.0%; best router Auto Code 82.3% |
| Terminal Bench 4.0 | 42.5% at $0.29/task | Auto Code 50.0% at $4.96/task |
| SWE Atlas QA | 60.5% at $0.14/task | Fugu Ultra v2 69.5% at $8.91/task |

Pareto has the top Router Index score on Deep SWE, SWE Atlas QA and Terminal Bench 4.0; the index counts cost and speed, so on SWE Atlas QA and Terminal Bench a more expensive router got the higher pass rate. Auto leads the index on Tau3 Banking and Auto Code on MMMU Pro Vision.

#### 4. What a day's budget buys on Deep SWE (estimate)

Finished tasks per day = daily budget ÷ cost per success. Benchmark tasks aren't your tasks; this is for scale, not a forecast.

| Row | Cost / success | $200 a day buys | $100 a day buys |
| --- | --- | --- | --- |
| Claude Opus 5.5, $11.00 row | $11.00 | 18 | 9 |
| Claude Opus 5.5, $2.27 row | $2.27 | 88 | 44 |
| Kimi K3.1 Pro | $1.09 | 183 | 91 |
| GPT-6.1-Sol (row "GPT-6.1-Sol · openai", 72.6%) | $0.41 | 487 | 243 |
| Auto (Medium) | $0.83 | 240 | 120 |
| Pareto | $0.26 | 769 | 384 |

Math: 200 ÷ 0.41 = 487.8; 100 ÷ 0.41 = 243.9; 200 ÷ 11.00 = 18.2; 100 ÷ 11.00 = 9.1; 200 ÷ 2.27 = 88.1; 100 ÷ 2.27 = 44.1; 200 ÷ 1.09 = 183.5; 100 ÷ 1.09 = 91.7; 200 ÷ 0.83 = 241.0; 100 ÷ 0.83 = 120.5; 200 ÷ 0.26 = 769.2; 100 ÷ 0.26 = 384.6. Rounded down.

#### 5. Re-weighted index, cost per task, and the Jev row (2026-11-05)

Screenshots taken in a headless browser on 2026-11-05, saved next to the draft: `openrouter-index-default-2026-11-05.png`, `openrouter-index-reweighted-2026-11-05.png`, `openrouter-cost-per-task-2026-11-05.png`.

Model Routing Index, routers only, weights moved by hand to 35% quality / 30% speed / 35% cost (page sliders at 35 and 65):

| Router | Deep SWE | SWE Atlas QA | Terminal Bench 4.0 | Tau3 Banking | MMMU Pro Vision |
| --- | --- | --- | --- | --- | --- |
| Auto | 8.6 | 8.6 | 7.3 | 8.4 | 8.3 |
| Auto Code | 7.6 | 8.4 | 6.0 | 8.3 | 9.4 (best) |
| Fugu Max | 4.8 | 6.6 | — | 6.9 | 8.6 |
| Fugu Ultra v2 | — | 6.0 | — | 6.0 | 7.0 |
| Pareto | 9.8 (best) | 9.4 (best) | 8.2 (best) | 8.8 (best) | 8.9 |

Default weights (60/20/20): Pareto best on Deep SWE (9.9), SWE Atlas QA (9.1), Terminal Bench 4.0 (8.3); Auto best on Tau3 Banking (9.0); Auto Code best on MMMU Pro Vision (9.7).

Cost per task, routers only: Pareto $0.19 / $0.14 / $0.29 / $0.15 / $0.015; Auto $0.57 / $0.36 / $1.33 / $0.60 / $0.020; Auto Code $1.98 / $0.66 / $4.96 / $0.60 / $0.025; Fugu Max $3.00 / $1.56 / — / $0.59 / $0.019; Fugu Ultra v2 — / $8.91 / — / $2.84 / $0.11. Pareto is lowest among routers on all five. The page's trophies also count Switchyard pairs: Gemini 4 Argon + Gemini 3.8 Flash ($0.083) on Terminal Bench 4.0 and DeepSeek v4.1 Flash + GPT-6-Sol ($0.011) on vision.

Jev: on 2026-10-29 the page listed "Jev Router (medium)" as its own row, leading MMMU Pro Vision (read for an earlier, unpublished draft). On 2026-11-05 the page has no Jev row and says "Auto Router is now powered by Jev Router."

#### 6. OpenRouter's launch, 2026-11-06: new routers and the redesigned board

Read 2026-11-06 in a headless browser; text saved in `research/openrouter-routers-2026-11-06.txt`, `research/openrouter-blog-auto-routers-2026-11-06.txt` and `research/openrouter-rankings-2026-11-06.txt`.

- **The launch.** OpenRouter's blog, "Introducing the New Auto and Auto Code Routers" (Brian Thomas, 2026-11-06), and its [launch thread](https://x.com/OpenRouter/status/2118740149106573670) (posted 16:19 UTC): new Auto Router and a new coding router, Auto Code; "eleven factors about each request"; routing reasons in the chatroom and via router metadata in the API; live routing mix on each router's page; guardrails, ZDR, provider restrictions, allowed lists and BYOK define the pool. The thread on routing reasons: "We provide detailed feedback via both our chatroom and api (by enabling router metadata) about why we selected a certain model. This includes the models considered, why they were selected, and the factors influencing the decision." (thread text as shared by Louis, saved in `research/openrouter-launch-thread-2026-11-06.txt`). Quotes used in the post: "Across the five benchmarks below, Auto Code scores in line with Claude Opus 5.5 while costing 74% less per task." "Benchmarks consistently show that no router on the market beats the single best frontier model at every point on the cost and quality curve. The new Auto Router is no exception." "Auto Router sees over 2.5T tokens from over 100k people every month."
- **The board.** "Last benchmark run Nov 5, 2026, 4:19 PM UTC" (was Nov 4, 4:38 PM). 152 published runs (was 151). Every router row on all five tests is unchanged: pass rates, cost per task, cost per success and the Model Routing Index at both 60/20/20 and 35/30/35 (re-read 2026-11-06, same values as section 5). New rows are single models and Switchyard pairs; on Deep SWE, Switchyard GPT-6.1-Sol + GPT-6-Luna (75.2%) now has the trophy on the quality view.
- **The layout.** The page now opens with the "Model Routing Frontier" chart and the matrix opens on the Quality tab; on 2026-11-05 it opened on the Model Routing Index. On Quality alone, router trophies go to Auto (Tau3 Banking 53.1%), Auto Code (Terminal Bench 4.0 50.0%) and Fugu Ultra v2 (SWE Atlas QA 69.5%); Switchyard pairs take Deep SWE and vision. Pareto leads none on raw quality.
- **The frontier** (quality score relative to the best router per test, averaged over five tests; cost per task): Auto Code 9.8 at $1.64; Auto (Medium tier) 9.3 at $0.58; Pareto 8.6 at $0.16; Fugu Ultra v2 8.8 at $3.95; Fugu Max 6.8 at $1.29. The chart's frontier line joins Pareto, Auto (Medium) and Auto Code. Pareto vs Auto Code: 0.16 ÷ 1.64 = 0.098 (about a tenth of the cost); 8.6 ÷ 9.8 = 0.878 (about 12% lower score). Single-model reference: Kimi K3.1 Pro 9.6 at $0.50.
- **Share of traffic.** Rankings, "This Month" (trailing 30 days through Nov 5), top ten models: 121 + 106 + 58.5 + 35 + 34.3 + 33.9 + 29.4 + 28.6 + 20.7 + 19.5 = 486.9T tokens. Auto Router's "over 2.5T" a month ÷ 486.9T = 0.51%, and total OpenRouter traffic is larger than the top ten, so the share is lower still. "Under 1%" holds unless Auto Router's monthly volume is above 4.9T. Estimate built from OpenRouter's own figures; OpenRouter doesn't publish Auto Router's exact share.
- **Screenshots** (2026-11-06): `openrouter-index-default-2026-11-06.png`, `openrouter-index-reweighted-2026-11-06.png` (sliders at 35 and 65), `openrouter-frontier-2026-11-06.png`. The post uses these three; the cost-per-task values are in section 5.

## Part C: Arena and Artificial Analysis

- Arena (arena.ai). Its blog index (https://arena.ai/blog/march-2026-arena-updates/, read 2026-11-05) lists "Introducing Max in Agent Arena," dated 12 Oct 2026: "Max, Arena's agentic router that achieves frontier performance in Agent Arena with 37% cost savings compared to top models." Its March update calls Max "our intelligent model router." We found no Arena ranking of other companies' routers. Confidence: verified for the quotes; "no ranking" is a search result, not proof of absence.
- Artificial Analysis, Coding Agent Index (https://artificialanalysis.ai/agents/coding-agents?routers=true), page data read 2026-11-05: 4 entries carry `"isRouter":true`: GitHub Copilot, Copilot Fusion (max); Devin Fusion CLI with GPT-6 Astra XHigh + SWE-2 Medium; Devin Fusion CLI with Claude Opus 5.5 Max + GPT-6 Luna High; Devin Fusion CLI with Claude Fable 5.1 XHigh + SWE-2 Medium. No standalone API routers (Pareto, Baseten, Ramp, OpenRouter Auto, Jev, Nexus Auto). Model pages for Jev and Pareto return 404. Confidence: verified.

## Limitations

- Vals is a commercial evaluation company. Vals ran each router at default settings, which may not be the setting a buyer would choose.
- OpenRouter runs its board and grades its own routers on it. It also distributes Pareto, and Stripe, an investor in our company, has agreed to buy OpenRouter. Shared, not neutral.
- One run per row as shown; the page doesn't show repeat counts or confidence intervals. Baseten's post makes a good case that single runs are noisy.
- Deep SWE on OpenRouter is not the same version as Baseten's DeepSWE 1.1, and Terminal Bench 4.0 isn't Terminal-Bench 2.1. Don't compare numbers across the two posts directly.
- Cost per success ignores the human's time on failures and retries.