{ "_about": "Live runs behind the 'Tested' box on each /use-cases/ page. The exact recipe code shown on the page is saved to files and run against https://api.unbiased.ai with model pareto. A run passes only if the script exits 0 and a separate checker confirms the printed records against the expected fields. Token counts and timings are what the API reported per request. Keys are redacted. A page without an entry here, or without result pass, makes no tested claim.", "checks": { "structured-extraction": { "date": "2026-10-20", "version": "Pydantic AI 2.58.0 and openai (Python) 3.35.1 on Python 3.15.0", "result": "pass", "served_model": "pareto", "response_model": "pareto-26.10", "protocol": "OpenAI Chat Completions with tool calling", "config": "tickets.py: OpenAIChatModel('pareto', provider=OpenAIProvider(base_url='https://api.unbiased.ai/v1', api_key=os.environ['UNBIASED_API_KEY'])); Agent(model, output_type=Ticket, instructions=...). tickets_openai.py: OpenAI(base_url='https://api.unbiased.ai/v1', api_key=os.environ['UNBIASED_API_KEY']).chat.completions.parse(model='pareto', tools=[pydantic_function_tool(Ticket)], tool_choice={'type': 'function', 'function': {'name': 'Ticket'}}). UNBIASED_API_KEY=", "task": "Extract a Ticket (customer, order_id, category, amount, urgent) from each of three messy support messages.", "checker": "Priya Nair: order_id A-10442, category shipping, urgent true. Tom Okafor: category billing, amount 96, order_id null. M. Lindqvist: category bug, urgent false, order_id and amount null.", "summary": "Both recipes turned three messy support messages into validated Ticket records with the right names, order number, amount and category, 3 runs out of 3 each.", "runs": 6, "records": 18, "retries": 0, "tokens_per_record": { "min": 371, "max": 694, "largest": { "input_tokens": 303, "output_tokens": 391 } }, "seconds_per_record": { "min": 0.91, "max": 2.45 }, "variants": { "pydantic-ai": { "result": "pass", "runs": 3, "passed": 3, "tokens_per_record": { "min": 371, "max": 520 } }, "openai-sdk": { "result": "pass", "runs": 3, "passed": 3, "tokens_per_record": { "min": 392, "max": 694 } } }, "notes": "Also tried on 2026-10-20: a JSON schema in response_format (openai chat.completions.parse(response_format=Ticket), and Pydantic AI NativeOutput) returned HTTP 400 unsupported_parameter; Pydantic AI PromptedOutput passed one run. Of the 12 runs whose raw output was saved that day, 1 returned urgent true for the double-charge message; every other field matched in all 12." }, "evaluate-pareto-on-your-data": { "date": "2026-10-20", "version": "promptfoo 0.124.1 (Node 26.11.1) and openai (Python) 3.35.1 on Python 3.15.0", "result": "pass", "served_model": "pareto", "response_model": "pareto-26.10", "protocol": "OpenAI Chat Completions", "config": "promptfooconfig.yaml: providers: [{id: openai:chat:pareto, label: Pareto, config: {apiBaseUrl: https://api.unbiased.ai/v1, apiKeyEnvar: UNBIASED_API_KEY, max_tokens: 4096}}], prompts: [file://prompt.txt], tests: file://cases.csv, defaultTest asserts a one-word answer; run with PROMPTFOO_DISABLE_TELEMETRY=1 npx promptfoo eval --no-cache. compare.py: OpenAI(base_url=\"https://api.unbiased.ai/v1\", api_key=os.environ[\"UNBIASED_API_KEY\"]).chat.completions.create(model=\"pareto\", ...) over the same prompt.txt and cases.csv. The current-model entry stays commented out in both: only Pareto was run. UNBIASED_API_KEY=", "task": "Label four support messages as billing, shipping, bug or other with one prompt template, and grade each answer with an assertion.", "checker": "promptfoo assertions: icontains on the expected label from cases.csv plus a javascript check that the answer is one word. compare.py: the expected label appears in the lowercased answer. Expected labels: billing, shipping, bug, other.", "summary": "promptfoo graded four support-triage cases against Pareto and all four passed in 3 runs out of 3; the plain OpenAI SDK script got the same four labels in 3 runs out of 3. Only Pareto was run, so there is no comparison result.", "runs": 6, "records": 24, "retries": 0, "tokens_per_record": { "min": 67, "max": 147 }, "seconds_per_record": { "min": 1.1, "max": 3.5 }, "variants": { "promptfoo": { "result": "pass", "runs": 3, "passed": 3, "cases": 12, "tokens_per_record": { "min": 67, "max": 76, "largest": { "input_tokens": 73, "output_tokens": 3 } }, "total_tokens_per_run": 288 }, "openai-sdk": { "result": "pass", "runs": 3, "passed": 3, "cases": 12, "tokens_per_record": { "min": 70, "max": 147 } } }, "notes": "A first promptfoo run on 2026-10-20 used a shorter prompt that did not define the labels; Pareto answered billing for \"do you have a student discount?\", which the test expected as other, so that run had 3 of 4 passing. The prompt then gained a definition of each label (the prompt shown on the page) and every later run passed. promptfoo sends temperature 0 and its own output limit; compare.py sends neither, and its token counts were higher and varied more between runs. Token counts are what the API reported; seconds are wall time per request. A later run on 2026-10-20 of the FAQ command npx promptfoo eval --repeat 3 --no-cache passed 12 of 12 (4 cases x 3 repeats, 864 tokens)." }, "ai-code-review-in-ci": { "date": "2026-10-20", "version": "Claude Code 2.1.305, Codex CLI 0.162.1 and Continue CLI 1.5.47", "result": "pass", "served_model": "pareto", "protocol": "Anthropic Messages (Claude Code), OpenAI Responses (Codex), OpenAI Chat Completions (Continue)", "config": "Each tool ran the exact run: block shown on the page, in a clone of a two-commit Python repo with origin/main checked out on the PR branch, BASE_REF=main, isolated HOME. Claude Code: ANTHROPIC_BASE_URL=https://api.unbiased.ai ANTHROPIC_AUTH_TOKEN= ANTHROPIC_MODEL=pareto; git diff \"origin/$BASE_REF...HEAD\" | claude -p --bare --tools \"\" --setting-sources user --append-system-prompt \"Review this diff.\" > review.md. Codex: ~/.codex/config.toml model = \"pareto\", model_provider = \"unbiased\", [model_providers.unbiased] base_url = \"https://api.unbiased.ai/v1\", env_key = \"UNBIASED_API_KEY\", wire_api = \"responses\"; UNBIASED_API_KEY= codex review --base \"origin/$BASE_REF\" > review.md. Continue: $RUNNER_TEMP/continue.yaml with provider openai, model pareto, apiBase https://api.unbiased.ai/v1, apiKey ; cn -p --silent --readonly --config \"$RUNNER_TEMP/continue.yaml\" > review.md.", "task": "Review a pull request whose only change adds an expiry check to coupon_is_valid() in coupons.py that returns True when now > expires_at, so expired coupons count as valid.", "checker": "A run passes if the command exits 0 and review.md names coupons.py, says expired coupons are treated as valid, and says the branch should return False. Checked by reading all nine files.", "summary": "Claude Code (claude -p --bare --tools \"\" --setting-sources user), Codex and Continue each flagged the inverted expiry check in the pull request and said to return False, 3 runs out of 3 each. With a pull request that added .claude/settings.json pointing ANTHROPIC_BASE_URL at a local server, the --bare-only command sent the step's token to that server; with --tools \"\" --setting-sources user it sent nothing there.", "runs": 9, "records": 9, "retries": 0, "seconds_per_review": { "min": 3, "max": 26 }, "variants": { "claude-code": { "result": "pass", "runs": 3, "passed": 3, "seconds": { "min": 3, "max": 6 } }, "codex": { "result": "pass", "runs": 3, "passed": 3, "seconds": { "min": 13, "max": 26 } }, "continue": { "result": "pass", "runs": 3, "passed": 3, "seconds": { "min": 6, "max": 6 } } }, "notes": "The GitHub Actions workflows themselves were not run; the run: blocks were run locally with the same environment variables. No tool posted a comment; gh pr comment was not run. Continue's docs (docs.continue.dev/cli/headless-mode) say cn takes piped input as context, but cn 1.5.47 ignored piped stdin in our test (a secret word piped in, answer NONE, re-run with isolated HOME), so its prompt names the git diff command and the model ran git itself in read-only mode. One Codex review (1 of 3) also said unexpired coupons are rejected, which is wrong; the bug and the fix were right in all nine. One Continue review gave lines 6-7 instead of 7-8. The three Claude Code runs were re-run on 2026-10-20 after QA found that --bare alone still starts Claude Code with tools [\"Bash\",\"Edit\",\"Read\"] and still loads the checkout's .claude/settings.json; the Claude Code figures here are from that re-run with --tools \"\" --setting-sources user added (3 of 3 passing, 3 to 6 seconds), in a fresh clone of the same repo with an isolated HOME. Two extra runs of the new Claude Code command with --output-format json reported 406 and 525 tokens (input 259 each, no cache reads) for this 2-line diff; the earlier command, with tools, reported 3,242 and 5,418. Tokens were not measured for Codex or Continue. In 1 of 3 Codex runs (codex-3), codex review ran python3 -c that imported the pull request's coupons.py and printed True 500: Codex's read-only sandbox blocks file writes and network but still runs commands, including the PR's code. No tool edited a file. cn 1.5.47 --readonly (plan mode) excludes Edit, MultiEdit and Write but sets Bash to allow (dist/index.js plan-mode policy list), and cn's Bash has no sandbox, so it can run any command, including the PR's code, write files through the shell and reach the network; in our 3 Continue runs it used Bash only for git diff and git log on origin/main...HEAD, plus Read. Settings attack, Claude Code 2.1.305, 2026-10-20: the PR branch added .claude/settings.json and .claude/settings.local.json with env.ANTHROPIC_BASE_URL=http://127.0.0.1: (one variant also set a dummy ANTHROPIC_AUTH_TOKEN) and permissions allowing Bash, Edit, Write and Read; a local listener logged requests. The step set ANTHROPIC_BASE_URL=https://api.unbiased.ai and a stand-in token. Old command (--bare only): system/init tools [\"Bash\",\"Edit\",\"Read\"], and the listener got POST /v1/messages carrying the step's stand-in token (base-URL-only variant). New command: system/init tools [], mcp_servers [], the listener got 0 requests, and requests went to api.unbiased.ai (401 for the stand-in token). With the real key and the new command on that hostile branch, the review passed and the listener again got 0 requests. Codex repo config, Codex CLI 0.162.1: a PR-added .codex/config.toml setting model_provider to a provider at the local listener and redefining [model_providers.unbiased] base_url to it did not change the target; with a fresh HOME, codex review printed provider: unbiased, sent to https://api.unbiased.ai/v1/responses and the listener got 0 requests. Other keys in a repo .codex/config.toml were not tested. Claude Code authenticated with ANTHROPIC_AUTH_TOKEN under --bare in every run; Claude Code's headless docs list only ANTHROPIC_API_KEY or apiKeyHelper for --bare. The post step's key check (grep -qF \"$KEY\" review.md, exit 1 if found) was run locally with a fake key: it passed a clean review.md and stopped on one containing the key. Codex's Linux sandbox on ubuntu-latest was not exercised; local runs were on macOS." }, "team-chat-workspace": { "date": "2026-10-20", "version": "Open WebUI 0.12.3 (pip) on Python 3.12", "result": "pass", "served_model": "pareto", "response_model": "pareto-26.10", "protocol": "OpenAI Chat Completions, called by Open WebUI with retrieved passages from a Knowledge base", "config": "open-webui serve with OPENAI_API_BASE_URL=https://api.unbiased.ai/v1, OPENAI_API_KEY=, OPENAI_API_CONFIGS={\"0\": {\"enable\": true, \"model_ids\": [\"pareto\"]}}, ENABLE_OLLAMA_API=False, ENABLE_API_KEYS=True; default local embeddings (sentence-transformers/all-MiniLM-L6-v2). Auth: the admin signed up or signed in (session JWT) only to create an Open WebUI API key (POST /api/v1/auths/api_key, sk-); every later call, including the chat requests, sent that API key as the Bearer token. Knowledge base \"Team handbook\" with one Markdown file, attached per request via the Open WebUI API: POST /api/chat/completions {\"model\": \"pareto\", \"stream\": false, \"files\": [{\"type\": \"collection\", \"id\": }]}.", "task": "Answer three questions whose answers exist only in a made-up one-page team handbook: on-call stipend and rotation start, start time after a late page, approval for a $600 purchase.", "checker": "Each answer must contain: Q1 200, Monday and 09:00; Q2 11:00; Q3 500 and approval. /api/models must list pareto with no manual step. Citation [1] recorded per answer, not required.", "summary": "Open WebUI listed pareto from environment variables alone and, called with an Open WebUI API key, answered three questions from an uploaded team handbook with the right facts, 9 answers out of 9 over 3 runs; all 9 cited the handbook with [1].", "runs": 3, "records": 9, "retries": 0, "tokens_per_record": { "min": 846, "max": 1144, "largest": { "input_tokens": 672, "output_tokens": 472 } }, "seconds_per_record": { "min": 3.9, "max": 6.76 }, "variants": { "open-webui": { "result": "pass", "runs": 3, "passed": 3, "tokens_per_record": { "min": 846, "max": 1144 } } }, "notes": "Records are answers. Input tokens per answer were 672 to 714, mostly retrieved handbook passages. Re-run on 2026-10-20 with ENABLE_API_KEYS=True and an Open WebUI API key; an earlier run the same day sent the session JWT instead and also passed 9 of 9 (724 to 1,078 tokens per answer). Also tried on 2026-10-20: a workspace model on pareto with the knowledge base attached (native function calling). Over the API without a session, Open WebUI sent no knowledge and Pareto answered generically; with a session id, Open WebUI sent its builtin tools and Pareto returned tool calls for query_knowledge_files, but the non-browser API path does not run the tool loop, so that path is not counted as tested. https://api.unbiased.ai/v1/embeddings returned HTTP 404 unknown_url. LibreChat, Onyx and LobeHub were not run." }, "batch-code-migrations": { "date": "2026-10-20", "version": "Codex CLI 0.162.1, Factory Droid 0.245.0 and Claude Code 2.1.305 on Python 3.15.0", "result": "pass", "served_model": "pareto", "protocol": "OpenAI Responses (Codex), OpenAI Chat Completions (Droid), Anthropic Messages (Claude Code)", "config": "migrate-codex.sh: codex exec --cd --sandbox workspace-write --ephemeral --json < /dev/null, with ~/.codex/config.toml model = \"pareto\", model_provider = \"unbiased\", [model_providers.unbiased] base_url = \"https://api.unbiased.ai/v1\", env_key = \"UNBIASED_API_KEY\", wire_api = \"responses\". migrate-droid.sh: (cd && droid exec --auto medium --model custom:Pareto-0 -o json ), with ~/.factory/settings.json customModels [{model: \"pareto\", displayName: \"Pareto\", baseUrl: \"https://api.unbiased.ai/v1\", apiKey: \"${UNBIASED_API_KEY}\", provider: \"generic-chat-completion-api\", maxOutputTokens: 32000}]. migrate-claude.sh: (cd && claude -p --permission-mode acceptEdits --allowedTools \"Bash(python3 *)\" --output-format json), with ANTHROPIC_BASE_URL=https://api.unbiased.ai ANTHROPIC_AUTH_TOKEN= ANTHROPIC_MODEL=pareto and the three ANTHROPIC_DEFAULT_*_MODEL=pareto. UNBIASED_API_KEY=", "task": "Three small Python git repos (billing, auth, reports) call the deprecated datetime.utcnow() and, in reports, datetime.utcfromtimestamp(), each with a different import style. Their unittest suites fail before the change under python3 -W error::DeprecationWarning. Each loop script runs its tool once per repo with the same prompt, then commits the repo only if the tests pass and no deprecated call is left.", "checker": "After each run, a separate script confirmed for every repo: the last commit is \"Use timezone-aware UTC datetimes\", it changes only the module file (tokens.py, invoice.py or daily.py) and no test file, python3 -W error::DeprecationWarning -m unittest exits 0, and no utcnow or utcfromtimestamp remains in any .py file.", "summary": "Each loop script made the same datetime migration in three Python repos and committed all three with tests passing, 3 runs out of 3 for Codex, Droid and Claude Code.", "runs": 9, "repos": 27, "retries": 0, "tokens_per_repo": { "min": 33860, "max": 95230, "note": "input plus output as each tool reported it, summed over the whole session; cached input included (Codex 23,509 to 55,921, Droid 29,212 to 44,458, Claude Code 38,812 to 61,098 per repo)" }, "seconds_per_loop": { "min": 24, "max": 41, "note": "wall time for all three repos, one after another" }, "seconds_per_repo": { "min": 6.0, "max": 16.6, "note": "duration_ms as reported by Droid (5.97 to 8.84) and Claude Code (7.15 to 16.62); Codex does not report it" }, "turns_per_repo": { "droid": { "min": 4, "max": 6 }, "claude-code": { "min": 5, "max": 12 } }, "variants": { "codex": { "result": "pass", "runs": 3, "passed": 3, "tokens_per_repo": { "min": 35907, "max": 65651 } }, "droid": { "result": "pass", "runs": 3, "passed": 3, "tokens_per_repo": { "min": 33860, "max": 51955 } }, "claude-code": { "result": "pass", "runs": 3, "passed": 3, "tokens_per_repo": { "min": 53860, "max": 95230 } } }, "notes": "All three tools wrote the same diffs: datetime.now(timezone.utc) or datetime.datetime.now(datetime.timezone.utc), dt.fromtimestamp(unix_ts, timezone.utc), and the timezone import where needed. Codex printed \"Model metadata for pareto not found. Defaulting to fallback metadata\" and Claude Code printed its unrecognized-model note; both finished normally. droid exec --cwd repos/auth/ failed with \"Working directory does not exist\" on a relative path, so the Droid script changes into the repo instead. An earlier Codex run without < /dev/null also passed 3 of 3; Codex prints \"Reading additional input from stdin\" in every run, with or without < /dev/null; the script redirects stdin from /dev/null so stdin ends at once and Codex cannot wait on an open pipe. In one Claude Code run a git diff call was denied by the allow list; the run still finished. Droid ran without a Factory login. The API key does not appear in any log or output. Codex appended a [projects.\"\"] trust_level = \"trusted\" entry to config.toml for each repo it ran in." }, "always-on-assistant": { "date": "2026-10-20", "version": "OpenClaw 2026.10.5", "result": "pass", "served_model": "pareto", "response_model": "pareto", "protocol": "OpenAI Chat Completions (OpenClaw provider api openai-completions)", "config": "openclaw.json: models.providers.unbiased = { baseUrl: \"https://api.unbiased.ai/v1\", apiKey: \"${UNBIASED_API_KEY}\", api: \"openai-completions\", models: [{ id: \"pareto\", name: \"Pareto\", contextWindow: 1000000, maxTokens: 131072 }] }; agents.defaults.model.primary = \"unbiased/pareto\"; agents.defaults.workspace = ; gateway = { mode: \"local\", bind: \"loopback\", port: 18999, auth: { mode: \"none\" } } for the scheduled runs. One-shot: openclaw agent exec --model unbiased/pareto --cwd ./proj --json --message-file task.md. Scheduled: openclaw gateway run, then openclaw cron add \"* * * * *\" \"\" --name \"Inbox triage\" --session isolated --model unbiased/pareto --no-deliver (third run also with --light-context). UNBIASED_API_KEY=", "task": "Read inbox.md (five made-up messages: a client outage, an accountant deadline, a gift-card scam, an FYI and a lunch question), sort each into urgent, reply, fyi or junk, draft replies for urgent and reply only, write triage.md and send nothing.", "checker": "triage.md must have 5 rows: 1 urgent, 2 reply, 3 junk, 4 fyi, 5 reply; a draft on rows 1, 2 and 5 and \"-\" on rows 3 and 4.", "summary": "OpenClaw sorted five made-up messages into urgent, reply, fyi and junk, drafted the three replies and sent nothing, 6 runs out of 6: 3 one-shot runs and 3 runs fired by its scheduler.", "runs": 6, "records": 30, "retries": 0, "tokens_per_pass": { "min": 28115, "max": 46951, "note": "input plus output plus cached input, as OpenClaw reported per pass of five messages", "one_shot": { "min": 28115, "max": 37589 }, "scheduled": { "min": 36817, "max": 46951 }, "scheduled_light_context": 36817, "cache_read_tokens": { "min": 16526, "max": 31189 }, "largest": { "input_tokens": 14911, "cache_read_tokens": 31189, "output_tokens": 851 } }, "seconds_per_pass": { "one_shot_wall": { "min": 11, "max": 16 }, "scheduled_duration": { "min": 7.6, "max": 11.1 } }, "variants": { "openclaw": { "result": "pass", "runs": 6, "passed": 6 }, "hermes-agent": { "result": "not run", "note": "Settings from Hermes Agent docs and hermes cron create --help (v0.22.2); not run for this job." }, "n8n": { "result": "not run", "note": "Settings from n8n docs; not run." } }, "notes": "Delivery was off (--no-deliver; delivery status not-requested) and no channel was connected, so nothing left the machine except model requests. Scheduled runs used a test schedule of every minute; the page shows every 30 minutes. inbox.md was a fixed file; fetching real mail into it was not tested. A first try with an empty key put the provider in a one-minute cooldown; it was rerun with a fresh state directory. Drafts were asked to use commas instead of dashes. Recheck on 2026-10-20 with the page's openclaw.json copied unchanged (gateway = { mode: \"local\" } only, default bind and port, ~/inbox workspace) into an isolated HOME: openclaw doctor exited 0, openclaw gateway run reached ready, the page's cron add \"*/30 * * * *\" ... --no-deliver exited 0, cron run wrote triage.md with all five buckets correct, and cron runs showed status ok, model pareto, delivered false, delivery not-requested. Without the gateway key, openclaw gateway run exits 78 (\"existing config is missing gateway.mode\")." }, "ci-autofix": { "date": "2026-10-20", "version": "Codex CLI 0.162.1 and Factory Droid 0.245.0 on Python 3.15.0", "result": "pass", "served_model": "pareto", "protocol": "OpenAI Responses (Codex) and OpenAI Chat Completions (Droid)", "config": "Codex: ~/.codex/config.toml with model = \"pareto\", model_provider = \"unbiased\", [model_providers.unbiased] base_url = \"https://api.unbiased.ai/v1\", env_key = \"UNBIASED_API_KEY\", wire_api = \"responses\", [shell_environment_policy] ignore_default_excludes = false; autofix-codex.sh builds PROMPT in three lines and runs codex exec -s workspace-write --json \"$PROMPT\" < /dev/null > codex.jsonl. Droid: ~/.factory/settings.json customModels [{model: \"pareto\", displayName: \"Pareto\", baseUrl: \"https://api.unbiased.ai/v1\", apiKey: \"${UNBIASED_API_KEY}\", provider: \"generic-chat-completion-api\", maxOutputTokens: 32000}]; autofix-droid.sh builds the same PROMPT and runs droid exec --auto medium --model \"custom:Pareto-0\" -o json \"$PROMPT\" > droid.json. Both scripts then run python3 -m unittest, git diff --exit-code -- tests and git diff > autofix.patch. UNBIASED_API_KEY=, set only for the script.", "task": "A git repo whose paginate() treats 1-based page numbers as 0-based, so python3 -m unittest fails 2 of 6 tests. The CI step saves the output to ci.log; the autofix script is then run with the prompt: CI failed: python3 -m unittest exited 1 and its output is in ci.log. Find the cause and fix the code, not the tests. Then run python3 -m unittest and confirm every test passes.", "checker": "Each run starts from a fresh copy of the failing commit with an isolated HOME. A run passes only if the autofix script exits 0 and, with the key unset, a separate rerun of python3 -m unittest passes all 6 tests, git diff shows no change under tests/, and the only changed file is shop/pages.py.", "summary": "Both tools read the CI log, changed one line in shop/pages.py and left the tests alone, and all 6 tests passed on a clean rerun, 3 runs out of 3 each.", "runs": 6, "retries": 0, "fix": "start = page * per_page became start = (page - 1) * per_page in all 6 runs.", "tokens_per_run": { "min": 37688, "max": 55455, "largest": { "input_tokens": 54578, "cached_input_tokens": 47018, "output_tokens": 877 } }, "seconds_per_run": { "min": 7.7, "max": 16.7 }, "variants": { "codex": { "result": "pass", "runs": 3, "passed": 3, "tokens_per_run": { "min": 37688, "max": 48120 }, "seconds_per_run": { "min": 7.7, "max": 10.8 } }, "droid": { "result": "pass", "runs": 3, "passed": 3, "tokens_per_run": { "min": 55219, "max": 55455 }, "seconds_per_run": { "min": 15.8, "max": 16.7 }, "turns": 6 } }, "notes": "Re-run on 2026-10-20 after two script changes (prompt split across three shell lines, and < /dev/null on codex exec); the figures here are from that re-run, 6 of 6 passing, and an earlier run of the previous scripts also passed 6 of 6. Token counts are what each tool reported for the whole run, input plus output, including cached input (Codex input_tokens includes cached_input_tokens; Droid input_tokens plus cache_read_input_tokens). Seconds are wall time for the autofix script, including the rerun. Codex printed its model metadata warning for pareto and 'Reading additional input from stdin...' in every run; with < /dev/null it read end of input and went on. Probe on 2026-10-20: a command Codex ran saw UNBIASED_API_KEY with the default shell_environment_policy and did not with ignore_default_excludes = false; a command Droid ran at --auto medium saw it. droid exec ran locally with no Factory sign-in and no FACTORY_API_KEY set; Droid logged Factory 401 responses (missing authorization token) that did not stop the run, and Factory's docs say to set FACTORY_API_KEY, so the page's workflow passes it. All runs were local on macOS, where Codex's workspace-write sandbox uses Seatbelt. Neither GitHub Actions workflow on the page was run on GitHub, and Codex's bubblewrap sandbox on Linux was not tested; the Codex workflow's sysctl line copies openai/codex-action's setup step for GitHub-hosted Ubuntu runners.", "linux_tested": false } } }