# Set up the Invoice data extraction recipe with Pareto: instructions for the agent

You are an AI agent helping someone run "Turn invoices into ledger-ready fields" on Pareto, Unbiased's model. Follow these steps in order and stop at any mismatch.

- Human page: https://unbiased.ai/use-cases/invoice-data-extraction/
- Status: Tested live by Unbiased in October 2026. Run record: https://unbiased.ai/data/use-case-checks.json (entry "invoice-data-extraction")
- Tools: Python, Pydantic AI (https://unbiased.ai/works-with/pydantic-ai/agent.md)
- Getting a key: https://unbiased.ai/start/agent.md

## What this recipe does

Unlike the parent email recipe, this evaluation starts with locally converted digital PDFs and checks amounts against generator truth. The headline is normalized field accuracy, aggregated per invoice across repeats. The resampling unit is the invoice, not independent fields.

## Results we measured

headline: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

line_item_f1: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

schema_validity: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

planted_errors_caught: Pareto 6.0000 vs baseline 6.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 6.0000, sample sd 0.0000, range 6.0000 to 6.0000; baseline mean 6.0000, sample sd 0.0000, range 6.0000 to 6.0000; better 0, tie 40, worse 0; sign p 1.0000.

false_flags: Pareto 0.0000 vs baseline 0.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 0.0000, sample sd 0.0000, range 0.0000 to 0.0000; baseline mean 0.0000, sample sd 0.0000, range 0.0000 to 0.0000; better 0, tie 40, worse 0; sign p 1.0000.

invoice_id_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

document_type_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

vendor_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

email_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

invoice_date_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

due_date_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

currency_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

subtotal_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

tax_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

total_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

bank_code_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

purchase_order_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000.

measured on Pareto 26.10, October 2026

- Dataset: https://unbiased.ai/data/use-case-evals/invoice-data-extraction/fixture.jsonl
- Dataset sha256: `a28cb8a9eee11fb196185071b5001b0593e789d7ee4198e9c16a2a4e4702385e`
- Source and license: Digital invoice PDFs we generated with reportlab 5.1.0 and converted to text with markitdown 0.1.9; 8 layouts, 4 currencies, 2 credit notes, 6 planted total errors in the test split and 2 in dev; CC BY 4.0
- Source notes: https://unbiased.ai/data/use-case-evals/invoice-data-extraction/SOURCE.md
- Run record: https://unbiased.ai/data/use-case-evals/invoice-data-extraction/receipt.json

## What Pareto is

Pareto is Unbiased’s blended AI model: it selects frontier and open models for each request and blends their outputs into one answer. One key and one model string, `pareto`, over two documented API formats:

- OpenAI format: base URL https://api.unbiased.ai/v1 (Chat Completions at /v1/chat/completions; Responses at /v1/responses).
- Anthropic format: base URL https://api.unbiased.ai with no /v1 (tools and SDKs add /v1/messages themselves).

Plans: Personal $10/month (50M tokens a week, 20 requests per minute), Personal Max $100/month (600M tokens a week, 60 requests per minute), Team pay as you go with prepaid credits (no weekly token cap, 300 requests per minute) at $0.80 in, $0.03 cached and $3.20 out per million tokens. Details: https://unbiased.ai/pricing/. Personal plans are for one person and are not the zero data retention tier; zero data retention covers only API traffic designated ZDR (https://unbiased.ai/private-ai/). Plan selection does not itself authorize production use. Confirm commercial or production use in a separate written agreement with Unbiased before moving live traffic.

## Safety rules

- Never ask the user to paste their API key into chat, and never print, echo or log it. Refer to it only as `$UNBIASED_API_KEY`.
- The user sets `UNBIASED_API_KEY` themselves, in their shell profile, keychain or CI secret store. To get a key, send them to https://unbiased.ai/start/agent.md (signup at platform.unbiased.ai; a person approves new accounts).
- Explain every file you will change, show the diff, and ask before writing. A request to learn about this is not permission to change their machine.
- Back up every file before editing: `cp <file> <file>.bak-$(date +%Y%m%d%H%M%S)`. Merge into existing config; do not overwrite it.
- Never write the key into a file, a repository or a command you show. Use the environment reference the tool supports.
- Do not commit anything. Do not change project-level config unless the user asks.
- If something does not match what this runbook says, stop and report it. Do not invent workarounds.
- Run the recipe on data and code the user chooses. Ask before sending their files to the API.

## 0. Choose a variant

This recipe has 3 variants: Python evaluation, Pydantic AI, Measured baseline. Ask the user which one they use, or check which tools are installed. Use only that variant's section below.

## 1. Check the key against the API, before touching anything

```sh
test -n "$UNBIASED_API_KEY" && echo set || echo missing
```

If it prints `missing`, stop. Tell the user to get a key (https://unbiased.ai/start/agent.md), add `export UNBIASED_API_KEY=...` with their key to their shell profile themselves, and open a new shell. Wait for them.

Then send one tiny request:

```sh
curl -sS https://api.unbiased.ai/v1/chat/completions \
  -H "Authorization: Bearer $UNBIASED_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"pareto","max_tokens":16,"messages":[{"role":"user","content":"Reply with the single word ok"}]}'
```

A JSON reply with a completion means the key works. A 401 means the key is wrong or the account is not approved yet: stop and tell the user. Do not print the key while debugging.

## 2. Steps from the human page

1. **Choose a plan and create a key.** Choose a plan, then create an API key after a person approves the account. Keep it in `UNBIASED_API_KEY`, never in source files or chat.
2. **Install in a virtual environment.** Install `openai==3.40.0`, `markitdown[pdf]==0.1.9` and `tiktoken==0.14.0`. Download fixture.jsonl, run_eval.py, recipe.py, check.py, eval.json, SOURCE.md and LICENSE.txt from the dataset section into that environment.
3. **Convert a digital PDF locally.** Run `markitdown invoice.pdf > invoice.txt`. The published fixture already contains the converted text. Verify it before sending confidential invoice content to any service.
4. **Force one record call.** The request sends a tool schema, not `response_format`. Both the portable runner and SDK entry point reject extra calls, even if `parallel_tool_calls=false` is set.
5. **Verify the returned fields.** For a dev example, run `python recipe.py --item dev-00 > arguments.json`, then `python check.py --item dev-00 --arguments arguments.json`. The checker compares expected values, line items and arithmetic; it exits nonzero on a mismatch.
6. **Reproduce only with approval.** Start with `python run_eval.py --plan --arms pareto --limit 10`. Ask before making requests, then run `--generate --arms pareto --limit 10`. Unset keys and run `--rescore RUN`. A limited run is a smoke check, not the full measurement.

## 3. Files and commands

Everything below is exactly as on the human page, with the key taken from the environment.

### Variant: Python evaluation

Status: scored run.

These are the exact published files. The scored runner uses Python HTTP, not the SDK transport. recipe.py exposes the same payload through an OpenAI SDK entry point, tested on dev only.

**recipe.py**

```python
"""MIT. Forced-tool extraction; arithmetic is checked locally, not by a model."""
import json
import os
from pathlib import Path

FIELDS = ['invoice_id', 'document_type', 'vendor', 'email', 'invoice_date', 'due_date', 'currency', 'subtotal', 'tax', 'total', 'bank_code', 'purchase_order']
LINE_FIELDS = ['description', 'quantity', 'unit_price', 'amount']
TOOL = {'type': 'function', 'function': {
    'name': 'record', 'description': 'Extract the printed invoice, without correcting totals.',
    'parameters': {'type': 'object', 'properties': {
        **{name: {'type': ['string', 'null']} for name in FIELDS},
        'line_items': {'type': 'array', 'items': {'type': 'object', 'properties': {name: {'type': 'string'} for name in LINE_FIELDS}, 'required': LINE_FIELDS, 'additionalProperties': False}},
        'note': {'type': 'string', 'description': 'Briefly describe any absent field or unusual sign. Not scored.'},
    }, 'required': FIELDS + ['line_items', 'note'], 'additionalProperties': False},
}}


def request(item, config):
    return {'messages': [{'role': 'system', 'content': config['prompt']}, {'role': 'user', 'content': item['text']}],
            'tools': [TOOL], 'tool_choice': config['tool_choice'], 'max_tokens': config['max_tokens'],
            'reasoning_effort': config['reasoning_effort'], 'parallel_tool_calls': False}


def extract(item, config, client):
    response = client.chat.completions.create(model=os.environ.get('MODEL', 'pareto'), **request(item, config))
    calls = response.choices[0].message.tool_calls or []
    if len(calls) != 1 or calls[0].function.name != 'record':
        raise ValueError('Expected exactly one record call; extra calls are rejected')
    return json.loads(calls[0].function.arguments)


if __name__ == '__main__':
    from openai import OpenAI
    import argparse
    parser = argparse.ArgumentParser()
    parser.add_argument('--item', default='dev-00')
    args = parser.parse_args()
    here = Path(__file__).resolve().parent
    config = json.loads((here / 'eval.json').read_text())
    item = next(json.loads(line) for line in (here / 'fixture.jsonl').read_text().splitlines() if json.loads(line)['item_id'] == args.item)
    model = os.environ.get('MODEL', 'pareto')
    client = OpenAI(base_url=os.environ.get('BASE_URL', 'https://api.unbiased.ai/v1'), api_key=os.environ['UNBIASED_API_KEY' if model == 'pareto' else 'OPENAI_API_KEY'], max_retries=0)
    print(json.dumps(extract(item, config, client), indent=2))
```

**check.py**

```python
"""MIT. Normalize fields, match line items and check printed totals locally."""
import argparse
from collections import Counter
from decimal import Decimal, InvalidOperation
import json
from pathlib import Path
import unicodedata

FIELDS = ['invoice_id', 'document_type', 'vendor', 'email', 'invoice_date', 'due_date', 'currency', 'subtotal', 'tax', 'total', 'bank_code', 'purchase_order']
LINE_FIELDS = ['description', 'quantity', 'unit_price', 'amount']
NUMERIC = {'subtotal', 'tax', 'total', 'quantity', 'unit_price', 'amount'}


def normalize(key, value):
    if value is None:
        return None
    if not isinstance(value, str):
        raise ValueError('Fields must be strings or null')
    value = ' '.join(unicodedata.normalize('NFKC', value).split())
    if key in NUMERIC:
        result = Decimal(value)
        if not result.is_finite():
            raise ValueError('Nonfinite number')
        return result
    return value


def equal(key, actual, expected):
    try:
        if key in NUMERIC:
            # Currency symbols and thousands separators are format, not value.
            stripped = str(actual).replace('$', '').replace('€', '').replace('£', '').replace(',', '')
            return normalize(key, stripped) == normalize(key, expected)
        return normalize(key, actual) == normalize(key, expected)
    except (InvalidOperation, ValueError):
        return False


def schema_valid(arguments):
    if not isinstance(arguments, dict) or set(arguments) != set(FIELDS + ['line_items', 'note']):
        return False
    if any(not isinstance(arguments[k], str) for k in FIELDS if k != 'purchase_order'):
        return False
    if arguments['purchase_order'] is not None and not isinstance(arguments['purchase_order'], str):
        return False
    if arguments['document_type'] not in ('invoice', 'credit_note') or arguments['currency'] not in ('USD', 'EUR', 'GBP', 'CAD'):
        return False
    return isinstance(arguments['note'], str) and isinstance(arguments['line_items'], list) and all(isinstance(r, dict) and set(r) == set(LINE_FIELDS) and all(isinstance(v, str) for v in r.values()) for r in arguments['line_items'])


def total_flag(arguments):
    try:
        values = [normalize(k, arguments.get(k)) for k in ('subtotal', 'tax', 'total')]
        return None if None in values else values[0] + values[1] != values[2]
    except (InvalidOperation, ValueError):
        return None


def line_key(row):
    return tuple(normalize(k, row[k]) for k in LINE_FIELDS)


def score(item, arguments, config):
    arguments = arguments if isinstance(arguments, dict) else {}
    fields = {k: int(k in arguments and equal(k, arguments[k], item['gold'][k])) for k in FIELDS}
    expected = Counter(line_key(row) for row in item['line_items'])
    got = Counter()
    invalid = 0
    lines = arguments.get('line_items', [])
    if not isinstance(lines, list):
        lines = []
    for row in lines:
        try:
            got[line_key(row)] += 1
        except (KeyError, TypeError, ValueError, InvalidOperation):
            invalid += 1
    tp = sum((expected & got).values())
    flag = total_flag(arguments)
    return {'score': sum(fields.values()) / len(FIELDS), 'secondary': {
        'line_item_f1': {'tp': tp, 'fp': sum(got.values()) + invalid - tp, 'fn': sum(expected.values()) - tp},
        'schema_validity': {'score': int(schema_valid(arguments))},
        'planted_errors_caught': {'score': int(item['total_error'] and flag is True)},
        'false_flags': {'score': int(not item['total_error'] and flag is True)},
        **{k + '_exact': {'score': v} for k, v in fields.items()},
    }}


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description='Offline example check; exits nonzero on any mismatch')
    parser.add_argument('--fixture', type=Path, default=Path(__file__).with_name('fixture.jsonl'))
    parser.add_argument('--item', default='dev-00')
    parser.add_argument('--arguments', type=Path, required=True)
    args = parser.parse_args()
    item = next(json.loads(line) for line in args.fixture.read_text().splitlines() if json.loads(line)['item_id'] == args.item)
    result = score(item, json.loads(args.arguments.read_text()), {})
    print(json.dumps(result, indent=2))
    secondary = result['secondary']
    raise SystemExit(0 if result['score'] == 1 and secondary['schema_validity']['score'] == 1 and secondary['line_item_f1']['fp'] == secondary['line_item_f1']['fn'] == 0 and total_flag(json.loads(args.arguments.read_text())) == item['total_error'] else 1)
```

**eval.json**

```json
{
  "unit": "invoice",
  "aggregation": "median",
  "headline": {
    "metric": "mean",
    "higher_is_better": true,
    "unit": "fraction of 12 invoice fields exact per record"
  },
  "secondary": [
    {
      "id": "line_item_f1",
      "metric": "f1",
      "higher_is_better": true,
      "unit": "fraction",
      "aggregation": "summed_counts"
    },
    {
      "id": "schema_validity",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    },
    {
      "id": "planted_errors_caught",
      "metric": "count",
      "higher_is_better": true,
      "unit": "invoices of 6 planted (or 34 clean) per run",
      "aggregation": "median"
    },
    {
      "id": "false_flags",
      "metric": "count",
      "higher_is_better": false,
      "unit": "invoices of 6 planted (or 34 clean) per run",
      "aggregation": "median"
    },
    {
      "id": "invoice_id_exact",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    },
    {
      "id": "document_type_exact",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    },
    {
      "id": "vendor_exact",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    },
    {
      "id": "email_exact",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    },
    {
      "id": "invoice_date_exact",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    },
    {
      "id": "due_date_exact",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    },
    {
      "id": "currency_exact",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    },
    {
      "id": "subtotal_exact",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    },
    {
      "id": "tax_exact",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    },
    {
      "id": "total_exact",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    },
    {
      "id": "bank_code_exact",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    },
    {
      "id": "purchase_order_exact",
      "metric": "mean",
      "higher_is_better": true,
      "unit": "fraction of records",
      "aggregation": "median"
    }
  ],
  "repeats": 3,
  "seed": 42,
  "verdict_rule": {
    "bootstrap": "paired-items",
    "resamples": 10000,
    "confidence": 0.95,
    "prng": "mulberry32",
    "quantile": "type-7",
    "range": "absolute-difference-exceeds-wider-run-range",
    "label_tie": "json-lexicographic",
    "zero_division": 0,
    "degenerate_kappa": 1,
    "rounding": "half-away-from-zero",
    "decimals": 4
  },
  "record_shape": [
    "score"
  ],
  "prompt": "You extract printed invoice and credit-note fields into one record tool call. One document per call. Copy every value exactly as the document prints it, including currency signs, minus marks and leading zeros. Do not compute, convert or correct any amount; if the document's printed total disagrees with subtotal plus tax, print the document's total anyway and say so in note. invoice_id, vendor, email, invoice_date, due_date, currency, subtotal, tax, total, bank_code and purchase_order come straight from the printed text. document_type is invoice or credit_note. invoice_date and due_date keep the document's YYYY-MM-DD form. subtotal, tax, total, quantity, unit_price and amount are plain decimal numbers without currency symbols or thousands separators. line_items repeats the printed line-item table rows in document order, one row object each with description, quantity, unit_price and amount as printed. purchase_order is null when the document does not show one. note is a short string: mention a missing field, a total that does not add up, or an empty string when nothing is unusual. Call the tool exactly once.",
  "fixture_sha256": "a28cb8a9eee11fb196185071b5001b0593e789d7ee4198e9c16a2a4e4702385e",
  "recipe_sha256": "8d582fa9010236a9711ee38f2a6593ad708bc3559c85614507a785fe9537ed29",
  "checker_sha256": "f2e2da02a8e76df38ac2f040394eba2bbf1240e43c4bf6d64047100ad83315ea",
  "runner_sha256": "2ef5bd134dfaa1491d139676b7d2e5da4b8e29a3a3edd8ccbdb98d3076118e7e",
  "max_tokens": 4096,
  "reasoning_effort": "low",
  "tool_choice": {
    "type": "function",
    "function": {
      "name": "record"
    }
  },
  "tuning_repeats": 1,
  "fixture_source": {
    "source": "Digital invoice PDFs we generated with reportlab 5.1.0 and converted to text with markitdown 0.1.9; 8 layouts, 4 currencies, 2 credit notes, 6 planted total errors in the test split and 2 in dev",
    "license": "CC BY 4.0",
    "generators": []
  },
  "planning": {
    "paired_difference_sd": 0.08
  },
  "variants": {
    "pydantic-ai": {
      "runs": [
        {
          "repeat": 1,
          "split": "dev",
          "result": "pass"
        },
        {
          "repeat": 2,
          "split": "dev",
          "result": "pass"
        },
        {
          "repeat": 3,
          "split": "dev",
          "result": "pass"
        }
      ],
      "checker": "check.py",
      "files_sha256": {
        "recipe.py": "2876d4398e9a6dcedb5a99b461800585734495bb29c604acc556386d1fa1c685",
        "check.py": "6786258c322de280081a2b4ac7de936e6d767708f51778327a7a54993913b7fd",
        "prompt.py": "39536c04424625486c242b7450be130e16670fbfe75e17356780809a763e8559"
      }
    }
  }
}
```

**run_eval.py**

```python
#!/usr/bin/env python3
"""Generated eval runner. Code: MIT. Requires Python 3.10+; --plan needs tiktoken==0.14.0."""
import sys, types
from pathlib import Path
runtime = types.ModuleType("runtime")
sys.modules["runtime"] = runtime
exec(compile('"""Key-free eval primitives. Only allowlisted records may leave this module."""\nimport email.utils\nimport gzip\nimport hashlib\nimport types\nimport json\nimport math\nimport os\nfrom pathlib import Path\nimport re\nimport shutil\nimport time\nimport urllib.error\nimport urllib.request\nfrom datetime import datetime, timezone\n\nPARETO_URL = "https://api.unbiased.ai/v1"\nBASELINE_URL = "https://api.openai.com/v1"\nBASELINE_MODEL = "gpt-6.1-sol"\nMAX_RAW_BYTES = 64 * 1024 * 1024\nRAW_KEYS = frozenset("item_id arm repeat request_model response_model endpoint provider_served allow_fallbacks reasoning_effort max_tokens tool_choice tool_arguments finish_reason tokens response_id started_at errors invalid_tool_calls retries gold prediction score tp fp fn secondary latency_ms".split())\nSCORE_KEYS = frozenset("gold prediction score tp fp fn secondary".split())\nTOKEN_KEYS = ("input", "output", "cached", "reasoning")\nSECRET = re.compile(r"\\b(?:sk-(?:proj-|or-v1-)?[A-Za-z0-9_-]{16,}|xox[bcp]-[A-Za-z0-9-]{16,})")\n\n\ndef sha256(data):\n    return hashlib.sha256(data).hexdigest()\n\n\ndef json_bytes(value):\n    return (json.dumps(value, ensure_ascii=False, indent=2) + "\\n").encode()\n\n\ndef jsonl_bytes(rows):\n    return ("".join(json.dumps(r, ensure_ascii=False, separators=(",", ":"), allow_nan=False) + "\\n" for r in rows)).encode()\n\n\ndef read_rows(path):\n    data = Path(path).read_bytes()\n    if str(path).endswith(".gz"):\n        with gzip.open(path, "rb") as stream:\n            data = stream.read(MAX_RAW_BYTES + 1)\n    if len(data) > MAX_RAW_BYTES:\n        raise ValueError("raw/fixture exceeds size limit")\n    return [json.loads(line) for line in data.decode().splitlines() if line.strip()]\n\n\ndef approved_raw(row):\n    if set(row) - RAW_KEYS:\n        raise ValueError("raw contains unapproved fields")\n    if set(row.get("tokens", {})) != set(TOKEN_KEYS):\n        raise ValueError("raw usage shape mismatch")\n    t = row["tokens"]\n    if any(type(t[k]) is not int or t[k] < 0 for k in TOKEN_KEYS) or t["cached"] > t["input"] or t["reasoning"] > t["output"]:\n        raise ValueError("invalid usage tokens")\n    encoded = json.dumps(row, ensure_ascii=False, allow_nan=False)\n    if SECRET.search(encoded):\n        raise ValueError("raw contains a key-like string")\n    return row\n\n\nclass Limiter:\n    def __init__(self, rpm=18, clock=time.monotonic, sleep=time.sleep):\n        if not 0 < rpm <= 30:\n            raise ValueError("rpm must be positive and at most 30")\n        self.interval, self.clock, self.sleep = 60 / rpm, clock, sleep\n        self.next = 0\n\n    def wait(self):\n        delay = self.next - self.clock()\n        if delay > 0:\n            self.sleep(delay)\n        self.next = self.clock() + self.interval\n\n\ndef retry_after(value, now=None):\n    try:\n        return max(0, float(value))\n    except (TypeError, ValueError):\n        try:\n            when = email.utils.parsedate_to_datetime(value).timestamp()\n            return max(0, when - (time.time() if now is None else now))\n        except (TypeError, ValueError, OverflowError):\n            return None\n\n\ndef identity(arm, model, endpoint, response=None):\n    if arm == "baseline":\n        if model != BASELINE_MODEL or endpoint != BASELINE_URL:\n            raise ValueError("baseline request not allowlisted; OpenRouter is unverified and disabled")\n        if response is not None:\n            if not re.fullmatch(r"gpt-6\\.1-sol(?:-\\d{4}-\\d{2}-\\d{2})?", response.get("model", "")):\n                raise ValueError("baseline response model not allowlisted")\n            # Direct OpenAI is the provider attestation. A contradictory field\n            # still aborts, rather than silently relabeling a proxied response.\n            if response.get("provider", "OpenAI") != "OpenAI":\n                raise ValueError("baseline provider not OpenAI")\n    elif arm == "pareto":\n        if model != "pareto" or endpoint != PARETO_URL:\n            raise ValueError("Pareto request not allowlisted")\n        if response is not None and not re.fullmatch(r"pareto-\\d{2}\\.\\d{1,2}", response.get("model", "")):\n            raise ValueError("Pareto response release missing")\n    else:\n        raise ValueError("unknown arm")\n\n\ndef api_key(arm):\n    name = "UNBIASED_API_KEY" if arm == "pareto" else "OPENAI_API_KEY"\n    value = os.environ.get(name)\n    if not value and arm == "pareto":\n        path = Path.home() / ".claude/secrets/unbiased-api.key"\n        if path.is_file():\n            value = path.read_text().strip()\n    if not value:\n        raise ValueError("missing " + name)\n    return value\n\n\nclass NoRedirect(urllib.request.HTTPRedirectHandler):\n    def redirect_request(self, req, fp, code, msg, headers, newurl):\n        raise ValueError("API redirects are forbidden")\n\n\nclass Client:\n    def __init__(self, arm, model, endpoint, key, limiter, opener=None, sleep=time.sleep, max_retries=3):\n        identity(arm, model, endpoint)\n        self.arm, self.model, self.endpoint = arm, model, endpoint\n        self.key, self.limiter, self.sleep, self.max_retries = key, limiter, sleep, max_retries\n        self.opener = opener or urllib.request.build_opener(NoRedirect()).open\n\n    def call(self, payload):\n        allowed = {"model", "messages", "tools", "tool_choice", "max_tokens", "reasoning_effort", "parallel_tool_calls"}\n        if set(payload) - allowed:\n            raise ValueError("request contains unapproved fields")\n        if any(not isinstance(m, dict) or m.get("content") is not None and not isinstance(m["content"], str) for m in payload.get("messages", [])):\n            raise ValueError("only text messages are supported")\n        encoded = json.dumps(payload, ensure_ascii=False)\n        if self.key in encoded or SECRET.search(encoded):\n            raise ValueError("request contains a key-like string")\n        if payload.get("model", self.model) != self.model:\n            raise ValueError("recipe changed model")\n        body = dict(payload, model=self.model)\n        retries = 0\n        while True:\n            self.limiter.wait()\n            req = urllib.request.Request(self.endpoint + "/chat/completions", data=json_bytes(body), headers={"Authorization": "Bearer " + self.key, "Content-Type": "application/json"})\n            try:\n                with self.opener(req, timeout=120) as resp:\n                    data = resp.read(MAX_RAW_BYTES + 1)\n                    if len(data) > MAX_RAW_BYTES:\n                        raise ValueError("API response exceeds size limit")\n                    response = json.loads(data)\n                identity(self.arm, self.model, self.endpoint, response)\n                output = json.dumps(response, ensure_ascii=False)\n                if self.key in output or SECRET.search(output):\n                    raise ValueError("model output contains a key-like string")\n                return response, retries\n            except urllib.error.HTTPError as error:\n                if error.code not in (429, 500, 502, 503, 504) or retries >= self.max_retries:\n                    raise ValueError("API request failed (HTTP %s); body withheld" % error.code) from None\n                delay = retry_after(error.headers.get("Retry-After"))\n                error.close()\n                self.sleep(delay if delay is not None else min(60, 2 ** retries))\n                retries += 1\n            except (urllib.error.URLError, TimeoutError):\n                # Do not retry an ambiguous completed request automatically.\n                raise ValueError("API transport failed; inspect partial run before resuming") from None\n\n\ndef usage(response):\n    u = response.get("usage", {})\n    return {"input": u.get("prompt_tokens", 0), "output": u.get("completion_tokens", 0),\n            "cached": u.get("prompt_tokens_details", {}).get("cached_tokens", 0),\n            "reasoning": u.get("completion_tokens_details", {}).get("reasoning_tokens", 0)}\n\n\ndef score_record(item, arguments, checker, config):\n    scored = checker.score(item, arguments, config)\n    if not isinstance(scored, dict) or set(scored) - SCORE_KEYS:\n        raise ValueError("checker returned unapproved fields")\n    shape = {"majority_label": ("gold", "prediction"), "median": ("score",), "summed_counts": ("tp", "fp", "fn")}\n    def check(payload, aggregation):\n        required = shape[aggregation]\n        if set(payload) != set(required):\n            raise ValueError("checker record shape mismatch")\n        if aggregation == "median" and (type(payload["score"]) not in (int, float) or not math.isfinite(payload["score"])):\n            raise ValueError("checker score must be finite")\n        if aggregation == "summed_counts" and any(type(payload[k]) is not int or payload[k] < 0 for k in required):\n            raise ValueError("checker counts must be nonnegative integers")\n    check({k: v for k, v in scored.items() if k != "secondary"}, config["aggregation"])\n    if config["aggregation"] == "majority_label" and scored["gold"] != item["gold"]:\n        raise ValueError("checker changed frozen gold")\n    secondary = scored.setdefault("secondary", {})\n    if set(secondary) != {s["id"] for s in config.get("secondary", [])}:\n        raise ValueError("checker secondary metrics mismatch")\n    for spec in config.get("secondary", []):\n        check(secondary[spec["id"]], spec["aggregation"])\n    return scored\n\n\ndef generate_record(item, arm, repeat, config, recipe, client):\n    started = datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")\n    begin = time.monotonic()\n    calls = []\n    def call(payload):\n        if payload.get("max_tokens") != config["max_tokens"] or payload.get("reasoning_effort") != config["reasoning_effort"]:\n            raise ValueError("recipe changed frozen token limit or reasoning effort")\n        choices = [config["tool_choice"], *config.get("tool_steps", [])]\n        if payload.get("tool_choice") not in choices or payload.get("tool_choice", {}).get("type") != "function":\n            raise ValueError("recipe changed frozen forced tool choice")\n        if not isinstance(payload.get("tools"), list) or not payload["tools"]:\n            raise ValueError("recipe requires tools")\n        response, retries = client.call(payload)\n        calls.append((response, retries, payload["tool_choice"]))\n        return response\n    # generate may implement a dependent tool loop using this checked call.\n    if hasattr(recipe, "generate"):\n        recipe.generate(item, config, call)\n    else:\n        call(recipe.request(item, config))\n    if not calls:\n        raise ValueError("recipe did not call the model")\n    response = calls[-1][0]\n    final = calls[-1][2]\n    if final != config["tool_choice"]:\n        raise ValueError("loop must end on the frozen forced tool choice")\n    choice = response["choices"][0]\n    tools = choice.get("message", {}).get("tool_calls", [])\n    invalid = 0\n    arguments = {}\n    try:\n        if len(tools) != 1 or tools[0]["function"]["name"] != final["function"]["name"]:\n            raise ValueError("expected exactly one final named tool call")\n        arguments = json.loads(tools[0]["function"]["arguments"])\n        if not isinstance(arguments, dict):\n            raise ValueError("tool arguments must be an object")\n    except (ValueError, KeyError, TypeError):\n        arguments, invalid = {}, 1\n    models = {r["model"] for r, _, _ in calls}\n    if len(models) != 1:\n        raise ValueError("served model changed within item")\n    row = {"item_id": item["item_id"], "arm": arm, "repeat": repeat,\n           "request_model": client.model, "response_model": response["model"], "endpoint": client.endpoint,\n           "provider_served": "OpenAI" if arm == "baseline" else "Unbiased", "allow_fallbacks": False,\n           "reasoning_effort": config["reasoning_effort"], "max_tokens": config["max_tokens"], "tool_choice": config["tool_choice"],\n           "tool_arguments": arguments, "finish_reason": "length" if any(r["choices"][0]["finish_reason"] == "length" for r, _, _ in calls) else choice["finish_reason"],\n           "tokens": {k: sum(usage(r)[k] for r, _, _ in calls) for k in TOKEN_KEYS},\n           "response_id": ",".join(r["id"] for r, _, _ in calls), "started_at": started, "errors": 0,\n           "invalid_tool_calls": invalid, "retries": sum(n for _, n, _ in calls), "latency_ms": round((time.monotonic() - begin) * 1000)}\n    return approved_raw(row)\n\n\ndef load_module(path, name):\n    module = types.ModuleType(name)\n    module.__file__ = str(path)\n    exec(compile(Path(path).read_bytes(), str(path), "exec"), module.__dict__)\n    return module\n\n\ndef retain_runs(directory):\n    # Only directories created by this runner (with its marker) are eligible.\n    runs = sorted(p for p in directory.iterdir() if p.is_dir() and not p.is_symlink() and (p / "manifest.json").is_file())\n    for old in runs[:-2]:\n        manifest = json.loads((old / "manifest.json").read_text())\n        if manifest.get("format") != "uc-eval-run-v1":\n            raise ValueError("refusing to remove unrecognized run")\n        if set(p.name for p in old.iterdir()) - {"manifest.json", "raw.jsonl.gz", "scored.jsonl.gz", "receipt.json"}:\n            raise ValueError("refusing to remove run with unexpected files")\n        shutil.rmtree(old)\n\n\ndef write_raw(path, rows):\n    data = jsonl_bytes([approved_raw(row) for row in rows])\n    if len(data) > MAX_RAW_BYTES:\n        raise ValueError("raw log exceeds size limit")\n    Path(path).write_bytes(gzip.compress(data, mtime=0))\n', "runtime", "exec"), runtime.__dict__)
sandbox = types.ModuleType("sandbox")
sys.modules["sandbox"] = sandbox
exec(compile('"""Local macOS checker sandbox. No keys, network, child processes or host writes."""\nimport json\nimport os\nfrom pathlib import Path\nimport signal\nimport subprocess\nimport sys\nimport tempfile\nimport time\n\n\n# This worker is trusted code. Generated Python may execute only after the\n# sandbox and rlimits are active. Checker modules must use in-process unittest,\n# not subprocesses. Each score gets a fresh interpreter and empty working dir.\nWORKER = r\'\'\'\nimport json, resource, sys, types\nfrom pathlib import Path\ncpu, memory = int(sys.argv[1]), int(sys.argv[2])\nresource.setrlimit(resource.RLIMIT_CPU, (cpu, cpu))\n# macOS rejects a small RLIMIT_DATA for modern Python. RSS is supervised\n# by the parent; CPU/file/core/descriptor limits are still enforced here.\nresource.setrlimit(resource.RLIMIT_FSIZE, (1048576, 1048576))\nresource.setrlimit(resource.RLIMIT_CORE, (0, 0))\nresource.setrlimit(resource.RLIMIT_NOFILE, (64, 64))\npayload = json.loads(Path(\'input.json\').read_text())\nchecker = types.ModuleType(\'uc_checker\')\nchecker.__file__ = str(Path(\'check.py\').resolve())\nexec(compile(Path(\'check.py\').read_bytes(), \'check.py\', \'exec\'), checker.__dict__)\nresult = checker.score(payload[\'item\'], payload[\'arguments\'], payload[\'config\'])\nPath(\'result.json\').write_text(json.dumps(result, allow_nan=False))\n\'\'\'\n\n\ndef score(checker_path, item, arguments, config, *, cpu_seconds=5, memory_mb=256, wall_seconds=10):\n    if sys.platform != \'darwin\' or not Path(\'/usr/bin/sandbox-exec\').is_file():\n        raise RuntimeError(\'macOS sandbox-exec is required; no unsandboxed fallback\')\n    if cpu_seconds < 1 or memory_mb < 16 or wall_seconds <= 0:\n        raise ValueError(\'invalid sandbox limits\')\n    memory = memory_mb * 1024 * 1024\n    with tempfile.TemporaryDirectory(prefix=\'uc-sandbox-\') as directory:\n        work = Path(directory).resolve()\n        (work / \'check.py\').write_bytes(Path(checker_path).read_bytes())\n        (work / \'input.json\').write_text(json.dumps({\'item\': item, \'arguments\': arguments, \'config\': config}))\n        # No blanket file-read: this machine has credentials. Python and OS\n        # libraries are read-only; only this fresh directory is writable.\n        roots = sorted({str(Path(sys.base_prefix).resolve()), sys.base_prefix, \'/System/Library\', \'/usr/lib\', \'/Library/Apple\', \'/private/var/db/dyld\'})\n        profile = \'(version 1)(deny default)(allow process-exec)(allow sysctl-read)\'\n        profile += \'(allow file-read-metadata)(allow file-read* (literal "/"))\'\n        profile += \'(allow file-read* \' + \' \'.join(\'(subpath \' + json.dumps(p) + \')\' for p in roots + [str(work)]) + \')\'\n        profile += \'(allow file-read* (literal \' + json.dumps(str(Path(sys.executable).resolve())) + \') (literal "/dev/urandom") (literal "/dev/null"))\'\n        profile += \'(allow file-write* (subpath \' + json.dumps(str(work)) + \') (literal "/dev/null"))\'\n        # Default-deny includes network*, process-fork, mach lookup and host IO.\n        env = {\'PATH\': \'/usr/bin:/bin\', \'HOME\': str(work), \'TMPDIR\': str(work), \'LC_ALL\': \'C\', \'PYTHONDONTWRITEBYTECODE\': \'1\'}\n        command = [\'/usr/bin/sandbox-exec\', \'-p\', profile, str(Path(sys.executable).resolve()), \'-I\', \'-B\', \'-c\', WORKER, str(cpu_seconds), str(memory)]\n        proc = subprocess.Popen(command, cwd=work, env=env, stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, start_new_session=True)\n        deadline = time.monotonic() + wall_seconds\n        try:\n            while proc.poll() is None:\n                if time.monotonic() >= deadline:\n                    raise TimeoutError(\'sandbox wall-time limit\')\n                # macOS does not enforce RLIMIT_RSS; supervise resident memory\n                # too. Fork is denied, so there can be no uncounted descendants.\n                rss = subprocess.run([\'/bin/ps\', \'-o\', \'rss=\', \'-p\', str(proc.pid)], capture_output=True, text=True, timeout=1)\n                if rss.returncode == 0 and rss.stdout.strip() and int(rss.stdout.strip()) * 1024 > memory:\n                    raise MemoryError(\'sandbox resident-memory limit\')\n                time.sleep(0.02)\n            if proc.returncode != 0:\n                raise RuntimeError(\'sandbox checker failed\')\n            result = work / \'result.json\'\n            if result.is_symlink() or result.stat().st_size > 1048576:\n                raise ValueError(\'invalid sandbox result\')\n            return json.loads(result.read_text())\n        finally:\n            if proc.poll() is None:\n                os.killpg(proc.pid, signal.SIGKILL)\n            proc.wait(timeout=2)\n', "sandbox", "exec"), sandbox.__dict__)
app = types.ModuleType("uc_app")
exec(compile('"""CLI shared by the private runner and generated per-page downloads."""\nimport argparse\nfrom datetime import datetime, timezone\nimport json\nimport math\nimport os\nfrom pathlib import Path\nimport re\nimport subprocess\nimport sys\n\nfrom runtime import (BASELINE_MODEL, BASELINE_URL, PARETO_URL, SCORE_KEYS, TOKEN_KEYS,\n                     Client, Limiter, api_key, approved_raw, generate_record, identity,\n                     json_bytes, load_module, read_rows, retain_runs, score_record,\n                     sha256, write_raw)\n\n\ndef canon(root):\n    script = "import {ACQUISITION_PLANS} from \'./scripts/claims.mjs\';import {readFileSync} from \'node:fs\';console.log(JSON.stringify({personal:ACQUISITION_PLANS.personal,prices:JSON.parse(readFileSync(\'data/prices.json\')).models}))"\n    return json.loads(subprocess.check_output(["node", "--input-type=module", "-e", script], cwd=root, timeout=10))\n\n\ndef load_page(root, slug, published=None):\n    source = published or root / "scripts/evals" / slug\n    data = published or root / "data/use-case-evals" / slug\n    config = json.loads((source / "eval.json").read_text())\n    items = read_rows(data / "fixture.jsonl")\n    if not items or any(not isinstance(i.get("item_id"), str) or not i["item_id"] or i.get("split") not in ("dev", "test") or not isinstance(i.get("text"), str) for i in items):\n        raise ValueError("fixture needs item_id, dev/test split and text")\n    if len({i["item_id"] for i in items}) != len(items):\n        raise ValueError("duplicate fixture ids")\n    for key, file in (("fixture_sha256", data / "fixture.jsonl"), ("recipe_sha256", source / "recipe.py"), ("checker_sha256", source / "check.py")):\n        if config.get(key) != sha256(file.read_bytes()):\n            raise ValueError("frozen " + key + " mismatch")\n    if config.get("runner_sha256") != sha256((source / "run_eval.py").read_bytes()):\n        raise ValueError("frozen runner_sha256 mismatch; generate runner before freezing eval.json")\n    if type(config.get("max_tokens")) is not int or config["max_tokens"] < 1 or not config.get("reasoning_effort"):\n        raise ValueError("freeze max_tokens and reasoning_effort")\n    if config.get("tool_choice", {}).get("type") != "function" or not config["tool_choice"].get("function", {}).get("name"):\n        raise ValueError("freeze a forced named tool_choice")\n    return source, data, config, items\n\n\ndef token_counter():\n    try:\n        import tiktoken\n    except ImportError:\n        raise ValueError("--plan requires tiktoken (python3 -m pip install tiktoken==0.14.0); no character-count fallback") from None\n    encoding = tiktoken.get_encoding("o200k_base")\n    return lambda text: len(encoding.encode(text, disallowed_special=()))\n\n\ndef make_plan(slug, config, items, recipe, arms, repeats, split, limit, rpm, pricing, count=None):\n    count = count or token_counter()\n    selected = [i for i in items if i["split"] == split][:limit]\n    dev = [i for i in items if i["split"] == "dev"][:limit]\n    if not selected:\n        raise ValueError("no items in selected split")\n    def budget(rows, n):\n        inputs = outputs = requests = 0\n        for item in rows:\n            # A loop recipe must supply all planned calls, including expected\n            # tool context. These are estimates, never measured API usage.\n            payloads = recipe.plan_requests(item, config) if hasattr(recipe, "plan_requests") else [recipe.request(item, config)]\n            if not payloads:\n                raise ValueError("recipe planning returned no requests")\n            for payload in payloads:\n                serialized = json.dumps({k: v for k, v in payload.items() if k != "model"}, ensure_ascii=False, sort_keys=True)\n                inputs += count(serialized)\n                outputs += config["max_tokens"]\n                requests += 1\n        return {"requests": requests * n, "input": inputs * n, "output": outputs * n}\n    scored = budget(selected, repeats)\n    tuning = budget(dev, config.get("tuning_repeats", 1))\n    total = {k: scored[k] * 2 + tuning[k] for k in scored}\n    weekly = pricing["personal"]["weeklyMillions"] * 1e6\n    price = next(p for p in pricing["prices"] if p["id"] == "gpt-6-1-sol")\n    n = len(selected)\n    # Paired bounded item differences in [-1,1], worst-case SD=1. Normal\n    # approximation, alpha=.05 two-sided and 80% power. Not a bootstrap result.\n    # Nonlinear set metrics require a dev SD on the metric\'s own scale.\n    metric = config["headline"]["metric"]\n    sd = config.get("planning", {}).get("paired_difference_sd")\n    assumption = "dev paired-difference SD on headline scale"\n    if sd is None and metric in ("accuracy", "mean", "count"):\n        bounds = config.get("planning", {}).get("score_bounds", [0, 1] if metric == "accuracy" else None)\n        if bounds and len(bounds) == 2 and bounds[1] > bounds[0]:\n            sd, assumption = bounds[1] - bounds[0], "worst-case bounded paired differences"\n    mde = None if sd is None else (1.959964 + 0.841621) * sd / math.sqrt(n)\n    if mde is not None and metric == "count":\n        mde *= n\n    return {"slug": slug, "split": split, "items": n, "repeats": repeats, "arms": arms,\n            "tokenizer": "o200k_base; serialized messages/tools and request overhead estimate",\n            "per_arm_scored": scored, "per_arm_tuning": tuning, "per_arm_rerun": scored,\n            "requests": total["requests"] * len(arms), "tokens": (total["input"] + total["output"]) * len(arms),\n            "output_assumption": "max_tokens per request including reasoning; cache savings not assumed; retries extra",\n            "personal_week_share_percent": (total["input"] + total["output"]) / weekly * 100 if "pareto" in arms else 0,\n            "personal_weekly_tokens": weekly, "rpm": rpm, "minutes": total["requests"] * len(arms) / rpm,\n            "baseline_cost_usd": (total["input"] * price["input_usd_per_mtok"] + total["output"] * price["output_usd_per_mtok"]) / 1e6 if "baseline" in arms else 0,\n            "minimum_detectable_difference": mde,\n            "mde_assumptions": assumption + "; approximate 95% confidence, 80% power, paired items not repeats; verdict also must exceed observed run ranges" if mde is not None else "not estimable for nonlinear headline without planning.paired_difference_sd from dev data; accept likely tie or supply SD"}\n\n\ndef checked_manifest(run, source, data):\n    manifest = json.loads((run / "manifest.json").read_text())\n    if manifest.get("format") != "uc-eval-run-v1":\n        raise ValueError("unknown run format")\n    for name, path in (("fixture.jsonl", data / "fixture.jsonl"), ("eval.json", source / "eval.json"), ("recipe.py", source / "recipe.py"), ("check.py", source / "check.py"), ("run_eval.py", source / "run_eval.py")):\n        if manifest["hashes"][name] != sha256(path.read_bytes()):\n            raise ValueError("run inputs changed: " + name)\n    if manifest.get("status") != "complete":\n        raise ValueError("run incomplete; inspect partial evidence before new generation")\n    if manifest["raw_sha256"] != sha256((run / "raw.jsonl.gz").read_bytes()):\n        raise ValueError("raw hash mismatch")\n    return manifest\n\n\ndef rescore(run, source, data, config, items):\n    if run.resolve().is_relative_to(data.resolve()) or run.resolve().is_relative_to(source.resolve()):\n        raise ValueError("rescore output must stay outside published evidence")\n    manifest = checked_manifest(run, source, data)\n    rows = read_rows(run / "raw.jsonl.gz")\n    selected = [i for i in items if i["split"] == manifest["split"]][:manifest["limit"]]\n    ids = {i["item_id"]: i for i in selected}\n    expected = {(i, a, r) for i in ids for a in manifest["arms"] for r in range(1, manifest["repeats"] + 1)}\n    seen = set()\n    if source.name == \'ai-unit-test-generation\' or config.get(\'sandbox\') == \'macos\':\n        from sandbox import score as sandbox_score\n        from types import SimpleNamespace\n        checker = SimpleNamespace(score=lambda item, arguments, config: sandbox_score(source / \'check.py\', item, arguments, config))\n    else:\n        checker = load_module(source / "check.py", "uc_checker")\n    scored = []\n    for row in rows:\n        approved_raw(row)\n        key = (row["item_id"], row["arm"], row["repeat"])\n        if key not in expected or key in seen:\n            raise ValueError("raw item/arm/repeat mismatch")\n        seen.add(key)\n        identity(row["arm"], row["request_model"], row["endpoint"], {"model": row["response_model"], "provider": row["provider_served"]})\n        if row["allow_fallbacks"] is not False or row["max_tokens"] != config["max_tokens"] or row["reasoning_effort"] != config["reasoning_effort"] or row["tool_choice"] != config["tool_choice"]:\n            raise ValueError("raw frozen request mismatch")\n        clean = {k: v for k, v in row.items() if k not in SCORE_KEYS}\n        clean.update(score_record(ids[row["item_id"]], row["tool_arguments"], checker, config))\n        scored.append(clean)\n    if seen != expected:\n        raise ValueError("raw missing items")\n    write_raw(run / "scored.jsonl.gz", scored)\n    return manifest, scored\n\n\ndef generate(slug, source, data, config, items, args, client_factory=None):\n    selected = [i for i in items if i["split"] == args.split][:args.limit]\n    if not selected:\n        raise ValueError("no items in selected split")\n    recipe = load_module(source / "recipe.py", "uc_recipe")\n    limiter = Limiter(args.rpm)\n    factory = client_factory or (lambda arm: Client(arm, "pareto" if arm == "pareto" else BASELINE_MODEL, PARETO_URL if arm == "pareto" else BASELINE_URL, api_key(arm), limiter))\n    clients = {arm: factory(arm) for arm in args.arms}\n    directory = Path(args.out).resolve() / slug\n    if directory.is_relative_to(data.resolve()) or directory.is_relative_to(source.resolve()) or \'data/use-case-evals\' in directory.as_posix():\n        raise ValueError("generation output must stay outside published evidence")\n    directory.mkdir(parents=True, exist_ok=True)\n    run = directory / datetime.now(timezone.utc).strftime("%Y-%m-%dT%H-%M-%S-%fZ")\n    run.mkdir()\n    manifest = {"format": "uc-eval-run-v1", "slug": slug, "status": "incomplete", "split": args.split,\n                "limit": args.limit, "arms": args.arms, "repeats": args.repeats,\n                "hashes": {name: sha256(path.read_bytes()) for name, path in (("fixture.jsonl", data / "fixture.jsonl"), ("eval.json", source / "eval.json"), ("recipe.py", source / "recipe.py"), ("check.py", source / "check.py"), ("run_eval.py", source / "run_eval.py"))}}\n    if getattr(args, \'freeze_sha\', None):\n        manifest[\'freeze_sha\'] = args.freeze_sha\n    (run / "manifest.json").write_bytes(json_bytes(manifest))\n    retain_runs(directory)\n    rows = []\n    try:\n        for repeat in range(1, args.repeats + 1):\n            for item in selected:\n                for arm in args.arms:\n                    rows.append(generate_record(item, arm, repeat, config, recipe, clients[arm]))\n                    # Persist each completed item so a timeout leaves evidence.\n                    write_raw(run / "raw.jsonl.gz", rows)\n        manifest["status"] = "complete"\n    finally:\n        if (run / "raw.jsonl.gz").exists():\n            manifest["raw_sha256"] = sha256((run / "raw.jsonl.gz").read_bytes())\n        (run / "manifest.json").write_bytes(json_bytes(manifest))\n    return run\n\n\ndef write_receipt(root, slug, source, data, run, manifest, rows, config, pending=False):\n    expected_arms = {\'pareto\'} if pending else {\'pareto\', \'baseline\'}\n    if manifest["split"] != "test" or set(manifest["arms"]) != expected_arms or manifest["limit"] is not None or manifest["repeats"] != config["repeats"] or config["repeats"] < (5 if slug == "llm-as-a-judge" else 3):\n        raise ValueError("receipt needs full paired frozen test repeats with no --limit")\n    git = lambda *args: subprocess.check_output(["git", *args], cwd=root, stderr=subprocess.PIPE, timeout=10)\n    freeze = manifest.get(\'freeze_sha\') or config.get("freeze_sha") or git("rev-parse", "HEAD").decode().strip()\n    git("merge-base", "--is-ancestor", freeze, "HEAD")\n    freeze_time = int(git("show", "-s", "--format=%ct", freeze))\n    for path in (source / "eval.json", source / "recipe.py", source / "check.py", source / "run_eval.py", data / "fixture.jsonl"):\n        if git("show", freeze + ":" + path.relative_to(root).as_posix()) != path.read_bytes():\n            raise ValueError("changed after freeze: " + path.name)\n    dates = {r["started_at"][:10] for r in rows}\n    if not dates or any(datetime.fromisoformat(r["started_at"].replace("Z", "+00:00")).timestamp() <= freeze_time for r in rows):\n        raise ValueError("receipt must follow freeze")\n    arms = {}\n    for arm in sorted(expected_arms):\n        subset = [r for r in rows if r["arm"] == arm]\n        models = {r["response_model"] for r in subset}\n        if len(models) != 1 or len({r["response_id"] for r in subset}) != len(subset) or any(r["errors"] for r in subset):\n            raise ValueError("run has errors, reused ids or changed model")\n        if not any(len({json.dumps(r["tool_arguments"], sort_keys=True) for r in subset if r["item_id"] == item}) > 1 for item in {r["item_id"] for r in subset}):\n            raise ValueError("repeat independence: identical outputs")\n        runs = []\n        for repeat in range(1, config["repeats"] + 1):\n            records = [r for r in subset if r["repeat"] == repeat]\n            runs.append({"repeat": repeat, "tokens": {k: sum(r["tokens"][k] for r in records) for k in TOKEN_KEYS},\n                         **{k: sum(r[k] for r in records) for k in ("errors", "invalid_tool_calls", "retries")},\n                         "truncations": sum(r["finish_reason"] == "length" for r in records),\n                         "response_ids_sha": sha256(json.dumps([r["response_id"] for r in records], ensure_ascii=False, separators=(",", ":")).encode())})\n        arms[arm] = {"model": subset[0]["request_model"], "endpoint": subset[0]["endpoint"], "max_tokens": config["max_tokens"], "runs": runs}\n        if arm == "pareto":\n            arms[arm]["response_model"] = subset[0]["response_model"]\n        else:\n            arms[arm].update(label="GPT-6.1 Sol", provider_served="OpenAI", reasoning_effort=config["reasoning_effort"])\n    fixture = read_rows(data / "fixture.jsonl")\n    prefix = data.relative_to(root).as_posix() + "/"\n    evaluation = {"schema": 2, "runner": (source / "run_eval.py").relative_to(root).as_posix(), "pareto_release": arms["pareto"]["response_model"],\n                  "fixture": {"path": prefix + "fixture.jsonl", "sha256": config["fixture_sha256"], "items": sum(i["split"] == "test" for i in fixture), "dev_items": sum(i["split"] == "dev" for i in fixture), "split": "test", **config["fixture_source"]},\n                  "prompt_sha256": sha256(config["prompt"].encode()), "recipe_sha256": config["recipe_sha256"], "eval_json_sha256": sha256((source / "eval.json").read_bytes()),\n                  "freeze_sha": freeze, **{k: config[k] for k in ("repeats", "seed", "unit", "aggregation", "headline", "secondary")}, "arms": arms, "variants": config.get("variants", {}),\n                  "raw": {"path": prefix + "runs/" + run.name + "/raw.jsonl.gz", "sha256": sha256((run / "scored.jsonl.gz").read_bytes())}}\n    if pending:\n        evaluation[\'baseline_status\'] = \'pending\'\n    receipt = {"date": max(dates), "result": "pass", "eval": evaluation}\n    candidate = run / "receipt.json"\n    candidate.write_bytes(json_bytes(receipt))\n    # Node owns schema/metric validation; validate before changing published data.\n    subprocess.run(["node", str(root / "scripts/evals/validate-receipt.mjs"), str(root), slug, str(candidate), str(run / "scored.jsonl.gz")], check=True, timeout=30, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n    target = root / evaluation["raw"]["path"]\n    target.parent.mkdir(parents=True, exist_ok=True)\n    target.write_bytes((run / "scored.jsonl.gz").read_bytes())\n    checks_path = root / "data/use-case-checks.json"\n    # Use JSON.stringify for byte identity with the existing builder/gate.\n    subprocess.run(["node", "-e", "const fs=require(\'fs\');const [p,s,r]=process.argv.slice(1);const d=JSON.parse(fs.readFileSync(p));const v=JSON.parse(fs.readFileSync(r));d.checks[s]=v;fs.writeFileSync(p,JSON.stringify(d,null,2)+\'\\\\n\');fs.writeFileSync(process.argv[4],JSON.stringify(v,null,2)+\'\\\\n\')", str(checks_path), slug, str(candidate), str(data / "receipt.json")], check=True, timeout=10)\n    # Keep only latest and previous published raw run, never other files.\n    previous = sorted(p for p in (data / "runs").iterdir() if p.is_dir() and not p.is_symlink())\n    for old in previous[:-2]:\n        if old != target.parent and {p.name for p in old.iterdir()} == {"raw.jsonl.gz"}:\n            (old / "raw.jsonl.gz").unlink()\n            old.rmdir()\n\n\ndef merge_baseline(root, slug, source, data, run, config, items):\n    receipt = json.loads((data / \'receipt.json\').read_text())\n    current = json.loads((root / \'data/use-case-checks.json\').read_text())[\'checks\'][slug]\n    if receipt != current or receipt[\'eval\'].get(\'baseline_status\') != \'pending\':\n        raise ValueError(\'merge requires the current pending receipt\')\n    e = receipt[\'eval\']\n    manifest, baseline = rescore(run, source, data, config, items)\n    if manifest[\'slug\'] != slug or manifest[\'arms\'] != [\'baseline\'] or manifest.get(\'freeze_sha\') != e[\'freeze_sha\']:\n        raise ValueError(\'baseline run must share the pending freeze sha\')\n    if manifest[\'split\'] != \'test\' or manifest[\'limit\'] is not None or manifest[\'repeats\'] != e[\'repeats\']:\n        raise ValueError(\'merge requires full frozen baseline repeats\')\n    for key, value in [(\'eval_json_sha256\', sha256((source / \'eval.json\').read_bytes())), (\'fixture\', config[\'fixture_sha256\'])]:\n        if (e[key][\'sha256\'] if key == \'fixture\' else e[key]) != value:\n            raise ValueError(\'pending frozen inputs changed\')\n    raw_path = (root / e[\'raw\'][\'path\']).resolve()\n    if not raw_path.is_relative_to(data.resolve()) or sha256(raw_path.read_bytes()) != e[\'raw\'][\'sha256\']:\n        raise ValueError(\'pending raw path/hash mismatch\')\n    pareto = read_rows(raw_path)\n    if any(r[\'arm\'] != \'pareto\' for r in pareto):\n        raise ValueError(\'pending raw contains another arm\')\n    # Revalidate the original receipt before combining, not just the new arm.\n    subprocess.run([\'node\', str(root / \'scripts/evals/validate-receipt.mjs\'), str(root), slug, str(data / \'receipt.json\'), str(raw_path)], check=True, timeout=30, stdout=subprocess.PIPE, stderr=subprocess.PIPE)\n    merged = dict(manifest, arms=[\'pareto\', \'baseline\'])\n    rows = pareto + baseline\n    write_raw(run / \'scored.jsonl.gz\', rows)\n    write_receipt(root, slug, source, data, run, merged, rows, config)\n\n\ndef main(root=None, published=None, pricing=None, slug_default=None):\n    root = Path(root).resolve() if root else None\n    parser = argparse.ArgumentParser(description="Two-phase use-case evals; no API calls in --plan or --rescore")\n    parser.add_argument("slug", nargs="?", default=slug_default)\n    parser.add_argument("--all", action="store_true")\n    modes = parser.add_mutually_exclusive_group()\n    modes.add_argument("--plan", action="store_true")\n    modes.add_argument("--generate", action="store_true")\n    modes.add_argument("--rescore", metavar="RUN")\n    modes.add_argument(\'--merge-baseline\', metavar=\'RUN\')\n    parser.add_argument(\'--pending\', action=\'store_true\')\n    parser.add_argument("--write-receipt", action="store_true")\n    parser.add_argument("--arms", help="default: both for planning, Pareto only for generation")\n    parser.add_argument("--repeats", type=int)\n    parser.add_argument("--split", choices=("dev", "test"), default="test")\n    parser.add_argument("--limit", type=int)\n    parser.add_argument("--rpm", type=float, default=18)\n    parser.add_argument("--out", default="/tmp/uc-smoke")\n    args = parser.parse_args()\n    if not (args.plan or args.generate or args.rescore or args.merge_baseline):\n        if args.write_receipt or args.arms:\n            args.generate = True\n        else:\n            parser.error(\'choose --plan, --generate, --rescore or --merge-baseline\')\n    if args.pending and (not args.write_receipt or args.plan or args.merge_baseline):\n        parser.error(\'--pending requires --write-receipt with generation or rescore\')\n    if args.merge_baseline and published:\n        parser.error(\'--merge-baseline requires the repository\')\n    args.arms = (args.arms or ("pareto,baseline" if args.plan else "pareto")).split(",")\n    if not args.arms or len(set(args.arms)) != len(args.arms) or set(args.arms) - {"pareto", "baseline"}:\n        parser.error("--arms must be pareto, baseline, or pareto,baseline")\n    if args.limit is not None and args.limit < 1 or args.repeats is not None and args.repeats < 1 or not 0 < args.rpm <= 30:\n        parser.error("positive limit/repeats and rpm <= 30 required")\n    if args.all and (not args.plan or args.slug and not published):\n        parser.error("--all is only supported for --plan without a slug")\n    if args.write_receipt and (args.plan or published):\n        parser.error("--write-receipt requires a repository generation or rescore")\n    if published and args.slug != slug_default:\n        parser.error("this download belongs to " + slug_default)\n    slugs = [args.slug] if args.slug else []\n    if args.all:\n        slugs = [slug_default] if published else sorted(p.parent.name for p in (root / "scripts/evals").glob("*/eval.json"))\n        if not slugs:\n            print("No eval pages yet. requests=0 tokens=0 baseline_cost_usd=0 minimum_detectable_difference=n/a")\n            return\n    if not slugs or any(not re.fullmatch(r"[a-z0-9]+(?:-[a-z0-9]+)*", s) for s in slugs):\n        parser.error("provide a safe slug or --all --plan")\n    for slug in slugs:\n        source, data, config, items = load_page(root, slug, published)\n        repeats = args.repeats or config["repeats"]\n        if args.plan:\n            recipe = load_module(source / "recipe.py", "uc_recipe")\n            print(json.dumps(make_plan(slug, config, items, recipe, args.arms, repeats, args.split, args.limit, args.rpm, pricing or canon(root)), indent=2))\n            continue\n        args.repeats = repeats\n        if args.merge_baseline:\n            merge_baseline(root, slug, source, data, Path(args.merge_baseline).resolve(), config, items)\n            print(\'Merged baseline; rebuild pages and run check-use-case-evals\')\n            continue\n        if args.pending and args.generate and args.arms != [\'pareto\']:\n            parser.error(\'--pending generates Pareto only\')\n        if root:\n            pending_path = data / \'receipt.json\'\n            prior = json.loads(pending_path.read_text()) if pending_path.exists() else {}\n            args.freeze_sha = prior.get(\'eval\', {}).get(\'freeze_sha\') if prior.get(\'eval\', {}).get(\'baseline_status\') == \'pending\' else None\n            args.freeze_sha = args.freeze_sha or config.get(\'freeze_sha\') or subprocess.check_output([\'git\', \'rev-parse\', \'HEAD\'], cwd=root, timeout=10).decode().strip()\n        if args.generate:\n            run = generate(slug, source, data, config, items, args)\n            print("Generated unscored run: " + str(run))\n            if not args.write_receipt:\n                continue\n        else:\n            run = Path(args.rescore).resolve()\n        manifest, rows = rescore(run, source, data, config, items)\n        print("Rescored without API calls: " + str(run / "scored.jsonl.gz"))\n        if args.write_receipt:\n            write_receipt(root, slug, source, data, run, manifest, rows, config, pending=args.pending)\n            print("Wrote receipt; rebuild pages and run check-use-case-evals before committing")\n\n\ndef entry(**kwargs):\n    try:\n        main(**kwargs)\n    except Exception as error:\n        # Provider bodies, recipe exception text and environment values can\n        # contain keys. CLI failures disclose only their class, not payloads.\n        print("Eval stopped: " + type(error).__name__ + "; check configuration, hashes, dependencies and partial run artifacts", file=sys.stderr)\n        return 1\n    return 0\n', "uc_app", "exec"), app.__dict__)
if __name__ == "__main__":
    sys.exit(app.entry(published=Path(__file__).resolve().parent, pricing={'personal': {'name': 'Personal', 'monthly': 10, 'weeklyMillions': 50, 'rpm': 20, 'campaign': 'pricing_personal', 'label': 'Start Personal'}, 'prices': [{'id': 'claude-fable-5-1', 'name': 'Claude Fable 5.1', 'provider': 'Anthropic', 'tier': 'premium', 'input_usd_per_mtok': 10, 'output_usd_per_mtok': 50, 'cache_read_multiplier': 0.025, 'cache_read_usd_per_mtok': 0.25, 'batch_discount': 0.5, 'source': 'https://platform.claude.com/docs/en/about-claude/pricing', 'verified': '2026-10-27', 'note': 'Same rate card as Fable 5, but cache reads bill at 0.025x base input ($0.25/MTok) instead of the standard 0.1x. Uses the newer tokenizer.'}, {'id': 'claude-fable-5', 'name': 'Claude Fable 5', 'provider': 'Anthropic', 'tier': 'premium', 'input_usd_per_mtok': 10, 'output_usd_per_mtok': 50, 'cache_read_multiplier': 0.1, 'cache_read_usd_per_mtok': 1, 'batch_discount': 0.5, 'source': 'https://platform.claude.com/docs/en/about-claude/pricing', 'verified': '2026-10-27'}, {'id': 'claude-opus-5-5', 'name': 'Claude Opus 5.5', 'provider': 'Anthropic', 'tier': 'frontier', 'input_usd_per_mtok': 4, 'output_usd_per_mtok': 20, 'cache_read_multiplier': 0.05, 'cache_read_usd_per_mtok': 0.2, 'cache_write_usd_per_mtok': 5, 'cache_write_multiplier': 1.25, 'batch_discount': 0.5, 'source': 'https://platform.claude.com/docs/en/about-claude/pricing', 'verified': '2026-10-27', 'note': "Verified 2026-10-27 on Anthropic's pricing page: $4 input / $20 output per MTok, 5-minute cache writes $5, 1-hour cache writes $8, cache reads $0.20 (0.05x input), batch $2 / $10. Full 1M context at standard pricing."}, {'id': 'claude-opus-5', 'name': 'Claude Opus 5', 'provider': 'Anthropic', 'tier': 'frontier', 'input_usd_per_mtok': 5, 'output_usd_per_mtok': 25, 'cache_read_multiplier': 0.1, 'cache_read_usd_per_mtok': 0.5, 'batch_discount': 0.5, 'source': 'https://platform.claude.com/docs/en/about-claude/pricing', 'verified': '2026-10-27'}, {'id': 'claude-opus-4-8', 'name': 'Claude Opus 4.8', 'provider': 'Anthropic', 'tier': 'frontier-prior', 'input_usd_per_mtok': 5, 'output_usd_per_mtok': 25, 'cache_read_multiplier': 0.1, 'cache_read_usd_per_mtok': 0.5, 'batch_discount': 0.5, 'source': 'https://platform.claude.com/docs/en/about-claude/pricing', 'verified': '2026-10-27'}, {'id': 'claude-sonnet-5-5', 'name': 'Claude Sonnet 5.5', 'provider': 'Anthropic', 'tier': 'mid', 'input_usd_per_mtok': 2, 'output_usd_per_mtok': 10, 'cache_read_multiplier': 0.05, 'cache_read_usd_per_mtok': 0.1, 'cache_write_usd_per_mtok': 2.5, 'cache_write_multiplier': 1.25, 'batch_discount': 0.5, 'source': 'https://platform.claude.com/docs/en/about-claude/pricing', 'verified': '2026-10-27', 'note': 'Verified 2026-10-27: $2 input / $10 output per MTok, cache reads $0.10 (0.05x input), 5-minute cache writes $2.50, batch 50% off. API model id claude-sonnet-5-5.'}, {'id': 'claude-sonnet-5', 'name': 'Claude Sonnet 5', 'provider': 'Anthropic', 'tier': 'mid', 'input_usd_per_mtok': 2, 'output_usd_per_mtok': 10, 'cache_read_multiplier': 0.1, 'batch_discount': 0.5, 'source': 'https://platform.claude.com/docs/en/about-claude/pricing', 'verified': '2026-10-27', 'note': '$2/$10 is now the standard rate. The increase to $3/$15 previously scheduled for 2026-09-01 was cancelled; Anthropic made the introductory price the standard price (platform release notes, August 10, 2026: https://platform.claude.com/docs/en/release-notes/overview).'}, {'id': 'claude-haiku-5-5', 'name': 'Claude Haiku 5.5', 'provider': 'Anthropic', 'tier': 'volume', 'input_usd_per_mtok': 0.1, 'output_usd_per_mtok': 0.5, 'cache_read_multiplier': 0.1, 'cache_read_usd_per_mtok': 0.01, 'cache_write_usd_per_mtok': 0.125, 'cache_write_multiplier': 1.25, 'batch_discount': 0.5, 'source': 'https://platform.claude.com/docs/en/about-claude/pricing', 'verified': '2026-10-27', 'note': "Verified 2026-10-27 on Anthropic's pricing page: $0.10 input / $0.50 output per MTok for prompts up to 100,000 tokens; $0.50 / $2.50 for prompts over 100,000 tokens. 5-minute cache writes $0.125, 1-hour $0.20, cache reads $0.01. Batch $0.05 / $0.25 (up to 100K)."}, {'id': 'claude-haiku-4-5', 'name': 'Claude Haiku 4.5', 'provider': 'Anthropic', 'tier': 'volume', 'input_usd_per_mtok': 1, 'output_usd_per_mtok': 5, 'cache_read_multiplier': 0.1, 'cache_read_usd_per_mtok': 0.1, 'batch_discount': 0.5, 'source': 'https://platform.claude.com/docs/en/about-claude/pricing', 'verified': '2026-10-27'}, {'id': 'gpt-6-1-sol', 'name': 'GPT-6.1 Sol', 'provider': 'OpenAI', 'tier': 'frontier', 'input_usd_per_mtok': 2, 'output_usd_per_mtok': 10, 'cache_read_usd_per_mtok': 0.1, 'cache_read_multiplier': 0.05, 'source': 'https://developers.openai.com/api/docs/pricing', 'verified': '2026-10-27', 'note': 'Verified 2026-10-27 against OpenAI Standard pricing, short context: $2 input / $0.10 cached input / $10 output per MTok. API model id gpt-6.1-sol.'}, {'id': 'gpt-6-luna', 'name': 'GPT-6 Luna', 'provider': 'OpenAI', 'tier': 'volume', 'input_usd_per_mtok': 0.1, 'output_usd_per_mtok': 0.5, 'cache_read_usd_per_mtok': 0.01, 'cache_read_multiplier': 0.1, 'source': 'https://developers.openai.com/api/docs/pricing', 'verified': '2026-10-27', 'note': 'Verified 2026-10-27 against OpenAI Standard pricing, short context: $0.10 input / $0.01 cached input / $0.50 output per MTok. API model id gpt-6-luna.'}, {'id': 'pareto', 'name': 'Pareto', 'provider': 'Unbiased', 'tier': 'blended', 'source': 'https://unbiased.ai/pricing/', 'verified': '2026-10-27', 'note': 'Published Pareto rates. Blended model: several LLMs per request, one synthesized answer. Benchmark receipts on the model card.', 'input_usd_per_mtok': 0.8, 'output_usd_per_mtok': 3.2, 'cache_read_usd_per_mtok': 0.03, 'cache_read_multiplier': 0.0375}, {'id': 'kimi-k3', 'name': 'Kimi K3', 'provider': 'Moonshot AI', 'tier': 'mid', 'input_usd_per_mtok': 3, 'output_usd_per_mtok': 15, 'cache_read_multiplier': 0.1, 'source': 'https://platform.kimi.ai/docs/pricing/chat', 'verified': '2026-10-27'}]}, slug_default='invoice-data-extraction'))
```

### Variant: Pydantic AI

Status: ran, not scored.

Dev-only smoke passes on a normal invoice, a credit note and a planted-total error. This is not the scored arm. The wrapper returns a validated dictionary; use the downloaded checker to compare it with dev gold.

**recipe.py**

```python
"""MIT. Pydantic AI smoke tab. Forced ToolOutput; text in, validated Invoice out."""
import json
import os
from pathlib import Path
from pydantic import BaseModel, Field
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider
from typing import Optional


class LineItem(BaseModel):
    description: str
    quantity: str
    unit_price: str
    amount: str


class Invoice(BaseModel):
    invoice_id: str
    document_type: str = Field(pattern='^(invoice|credit_note)$')
    vendor: str
    email: str
    invoice_date: str
    due_date: str
    currency: str = Field(pattern='^(USD|EUR|GBP|CAD)$')
    subtotal: str
    tax: str
    total: str
    bank_code: str
    purchase_order: str | None
    line_items: list[LineItem]
    note: str = ''


PROMPT = Path(__file__).with_name('prompt.py').read_text()


def extract(text):
    model = OpenAIChatModel('pareto', provider=OpenAIProvider(base_url='https://api.unbiased.ai/v1', api_key=os.environ['UNBIASED_API_KEY']))
    agent = Agent(model, output_type=Invoice, instructions=PROMPT)
    return agent.run_sync(text).output.model_dump(mode='json', exclude_none=True)
```

**prompt.py**

```python
PROMPT = "You extract printed invoice and credit-note fields into one record tool call. One document per call. Copy every value exactly as the document prints it, including currency signs, minus marks and leading zeros. Do not compute, convert or correct any amount; if the document's printed total disagrees with subtotal plus tax, print the document's total anyway and say so in note. invoice_id, vendor, email, invoice_date, due_date, currency, subtotal, tax, total, bank_code and purchase_order come straight from the printed text. document_type is invoice or credit_note. invoice_date and due_date keep the document's YYYY-MM-DD form. subtotal, tax, total, quantity, unit_price and amount are plain decimal numbers without currency symbols or thousands separators. line_items repeats the printed line-item table rows in document order, one row object each with description, quantity, unit_price and amount as printed. purchase_order is null when the document does not show one. note is a short string: mention a missing field, a total that does not add up, or an empty string when nothing is unusual. Call the tool exactly once."
```

**check.py**

```python
"""MIT. Smoke checker: exits 0 when the extracted Invoice matches dev gold fields."""
import argparse
import json
import sys
from pathlib import Path

FIELDS = ['invoice_id', 'document_type', 'vendor', 'email', 'invoice_date', 'due_date', 'currency', 'subtotal', 'tax', 'total', 'bank_code', 'purchase_order']


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--fixture', type=Path, required=True)
    parser.add_argument('--item', required=True)
    parser.add_argument('--arguments', type=Path, required=True)
    args = parser.parse_args()
    item = next(json.loads(line) for line in args.fixture.read_text().splitlines() if json.loads(line)['item_id'] == args.item)
    got = json.loads(args.arguments.read_text())
    wrong = [k for k in FIELDS if str(got.get(k)) != str(item['gold'][k])]
    print(json.dumps({'item_id': args.item, 'fields_wrong': wrong}))
    return 1 if wrong else 0


if __name__ == '__main__':
    sys.exit(main())
```

Note from the human page, word for word: Pydantic AI slim 2.62.0. Three repeats of dev-00, dev-07 and dev-02. The recipe has no command-line entry point; import extract(text). It reads prompt.py as text, including its assignment wrapper. These are the exact bytes tested, not a recommended prompt-file convention.

### Variant: Measured baseline

Status: card only; no runnable files.

Both arms used the frozen invoice text, prompt, forced record tool and checker. The repository runner pinned Azure with fallbacks disabled and omitted parallel_tool_calls because Azure rejects that parameter; extra calls were still rejected locally. The frozen downloadable runner predates this transport support and supports OpenAI direct only, so it cannot reproduce the Azure transport unchanged. After separate spend approval, its direct-provider check is `python run_eval.py --generate --arms baseline --repeats 3 --split test` with `OPENAI_API_KEY`. Never reuse the Unbiased key.



Note from the human page: Every invoice contributes the same number of fields. Unicode and whitespace are normalized; decimal formatting is normalized for numeric fields. Missing keys fail even when gold is null. The arithmetic check flags subtotal plus tax disagreements; it does not check tax law, payment authorization or credit-note line signs. Generated labels and easy repeated layouts limit generalization.

## 4. Verify with one real run

Run the variant's command once with `UNBIASED_API_KEY` set, on a small input, and show the user the output. Setup is complete only when a real result has come back from `pareto`.

Run the published `check.py` against the same selected fixture items and their expected gold labels or values, with keys unset. Inspect its CLI first and stop if it differs from the documented recipe. Report each mismatch and the checker exit status; a model reply alone is not verification.

## Reproduce our eval (optional; ask first)



The estimated budget for the full 3-repeat comparison is 240 requests across both arms, about 243,880 input plus output tokens, 0.32% of a Personal week for the Pareto arm, and $0.2729 Pareto vs $0.3307 baseline token cost at published rates. The default limited run covers at most 10 items per repeat: 60 requests across both arms, about 60,970 input plus output tokens, 0.08% of a Personal week for the Pareto arm, and $0.0682 Pareto vs $0.0827 baseline token cost at published rates. These estimates use measured tokens, include cache reads, exclude retries and can change with new outputs. Output tokens include reasoning. The Pareto request-rate floor uses 20 requests per minute.

Tell the user both budgets and ask before downloading files, installing dependencies or making any API request. Do not run the full comparison by default. A limited run is a smoke check, not a reproduction of the published score.

Keep baseline credentials separate: the user sets `OPENAI_API_KEY` themselves for OpenAI direct, just as they set `UNBIASED_API_KEY`. Never ask for either key in chat, print it, log it or write it into a file. Never copy the Pareto key into the baseline variable. This measured baseline was GPT-6.1 Sol, served by Azure through OpenRouter, with fallbacks disabled. Its key is `OPENROUTER_API_KEY`, also environment-only. The frozen published runner predates that transport support and defaults to direct OpenAI; it does not reproduce the measured provider. Do not substitute providers or claim an exact reproduction. Ask for a provider-pinned runner before making baseline requests.

After approval, download the published files into a new, empty folder:

- https://unbiased.ai/data/use-case-evals/invoice-data-extraction/run_eval.py
- https://unbiased.ai/data/use-case-evals/invoice-data-extraction/recipe.py
- https://unbiased.ai/data/use-case-evals/invoice-data-extraction/check.py
- https://unbiased.ai/data/use-case-evals/invoice-data-extraction/eval.json
- https://unbiased.ai/data/use-case-evals/invoice-data-extraction/SOURCE.md
- https://unbiased.ai/data/use-case-evals/invoice-data-extraction/LICENSE.txt
- Dataset: https://unbiased.ai/data/use-case-evals/invoice-data-extraction/fixture.jsonl
- Dataset sha256: `a28cb8a9eee11fb196185071b5001b0593e789d7ee4198e9c16a2a4e4702385e`
- Recipe sha256: `8d582fa9010236a9711ee38f2a6593ad708bc3559c85614507a785fe9537ed29`
- Eval config sha256: `f1ce9703f3ca0950b5882627a9018164f53d5bda7197ec27a91ffe97339756a6`

Verify the dataset hash before running. Read `eval.json` and `SOURCE.md` for the split, dependencies and any additional labels or files the recipe requires. Install only in a project virtualenv. Check `python3 run_eval.py --help` first. If its options differ, stop rather than inventing flags.

Default command after approval:

```sh
python3 -I run_eval.py --generate --arms pareto --repeats 3 --split test --limit 10
```

The recipe must use `MODEL` and `BASE_URL` from the selected arm's environment, with `pareto` and `https://api.unbiased.ai/v1` as the Pareto defaults. The measured baseline was `openai/gpt-6.1-sol` at `https://openrouter.ai/api/v1`. The frozen download instead uses `gpt-6.1-sol` at `https://api.openai.com/v1`; do not change its model to the measured OpenRouter ID, which it does not support. Use reasoning effort `low` and the same max_tokens as Pareto. Never send sampling parameters. Abort if the baseline response model does not match; do not substitute another provider or model.

Verify with the published `check.py` on the same selected items and labels, with keys unset. Check the checker's CLI before using it. Require a clean exit and compare each prediction with the expected gold values; report mismatches, invalid tool calls, truncations and retries. For a full run, recompute the headline and every secondary from raw records using the frozen `eval.json` rules; do not require Pareto to win.

## 5. Roll back

- Restore each `.bak-` file you made and delete the files you added.
- Remove any lines you added to the shell profile.

## 6. Troubleshooting and questions from the human page

- **Does the PDF itself go to Pareto?** No. markitdown converts it locally. Only extracted text goes into the request; layout and OCR are not model capabilities tested here.
- **How were the labels checked?** Labels set by the generator, spot-checked by us using automated checks before publication. Independent human spot-check is pending. No model generated the invoice text; SOURCE.md records the deterministic generator and license.
- **What does field accuracy measure?** Exact values after the frozen normalizations in check.py. Each invoice gets a fraction of matching fields, the median across repeats, and then invoices are averaged. Per-field rows are reported separately. This is not a measure of scan quality. Both arms reached the ceiling on this synthetic set; the tie does not establish equal real-world reliability.
- **What happens if the printed total is wrong?** Copy the printed amount unchanged, then flag the subtotal-plus-tax disagreement locally. The planted-error row counts flagged erroneous invoices; the false-flags row counts flagged clean invoices. Neither proves that a payment is valid.
- **Why force a tool instead of a response format?** The capability spike rejected `response_format` with a JSON schema. Forced calls worked. This recipe also rejects extra calls locally rather than assuming the parallel-call setting is enforced.
- **Can I use scanned invoices?** Not on the evidence here. The set contains digital PDFs we generated, not scans. Evaluate OCR separately before blaming extraction errors on the language model.
- **How is this different from the email extraction recipe?** This page publishes converted invoice text, intended field labels, arithmetic errors and per-field measurements. The parent recipe demonstrates fields extracted from support emails, not invoice PDF conversion.
- **Can I send confidential invoices?** Check your contract and retention requirements first. Ask about the Zero Data Retention API if the work must not be kept. This dataset contains synthetic records only, not customer invoices.
- **How does Pareto compare to GPT-6.1 Sol on this?** headline: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. line_item_f1: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. schema_validity: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. planted_errors_caught: Pareto 6.0000 vs baseline 6.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 6.0000, sample sd 0.0000, range 6.0000 to 6.0000; baseline mean 6.0000, sample sd 0.0000, range 6.0000 to 6.0000; better 0, tie 40, worse 0; sign p 1.0000. false_flags: Pareto 0.0000 vs baseline 0.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 0.0000, sample sd 0.0000, range 0.0000 to 0.0000; baseline mean 0.0000, sample sd 0.0000, range 0.0000 to 0.0000; better 0, tie 40, worse 0; sign p 1.0000. invoice_id_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. document_type_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. vendor_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. email_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. invoice_date_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. due_date_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. currency_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. subtotal_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. tax_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. total_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. bank_code_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. purchase_order_exact: Pareto 1.0000 vs baseline 1.0000; difference 0.0000; tie; 95% interval 0.0000 to 0.0000; Pareto mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; baseline mean 1.0000, sample sd 0.0000, range 1.0000 to 1.0000; better 0, tie 40, worse 0; sign p 1.0000. Baseline arm: GPT-6.1 Sol, served by Azure through OpenRouter. measured on Pareto 26.10, October 2026
- Unknown model ids are not rejected yet, so a typo is still served and billed as Pareto. Always send `pareto` exactly.
- OpenAI-format clients need /v1 in the base URL. Anthropic-format clients and SDKs take https://api.unbiased.ai without /v1.

## Production use

Plan selection does not itself authorize production use. Confirm commercial or production use in a separate written agreement with Unbiased before moving live traffic.

## Report back

Tell the user: the tool version, every file you changed with its backup path, the lines you added to their shell profile, the verify output (never the key) and anything you skipped. Problems go to info@unbiased.ai, with the failing step and the error text, never the key.