Fix the failing test before you look

Q: Will it push a fix to my branch?

Not with these files. Both scripts end by writing autofix.patch, the workflow uploads it as an artifact, and the build stays red. A person downloads the patch, reads it and applies it with git apply autofix.patch. Droid at --auto medium is not allowed to git push, and the workflow’s token only has contents: read.

Q: What stops the agent from editing the tests until they pass?

Two things. The prompt says to fix the code and not the tests, and the script does not trust the prompt: git diff --exit-code -- tests exits non-zero if anything under tests/ changed, so no patch is written. In all 6 of our runs the only changed file was shop/pages.py.

Q: Can the commands the agent runs see my API key?

With Codex, not if you keep ignore_default_excludes = false: in our check, a command Codex ran could not see UNBIASED_API_KEY with that line and could see it without it. With Droid, it could. Either way your tests and their dependencies run in the same job, so pass the key only to the autofix step and run this on your own code, not on untrusted pull requests.

Q: Can I use openai/codex-action instead of installing Codex myself?

The action’s inputs include responses-api-endpoint and model, so it may be pointed at https://api.unbiased.ai/v1/responses with model pareto. We have not tested that. The version on this page installs the CLI and uses the same config.toml we ran.

Q: Codex prints that model metadata for pareto was not found. Is the run broken?

No. Codex has metadata for OpenAI’s own model names and uses fallback metadata for any other name. It printed that warning in every one of our runs, which then fixed the bug and passed the rerun. Format compatibility is not a promise that every Codex or Droid feature works, so tell us if one doesn’t.

Q: How many fixes fit in a plan?

Personal includes 50M tokens a week at 20 requests a minute. Our fixes used about 38,000 to 56,000 tokens each, but a bigger repo or a harder failure can use several times that, so measure a few of your own before you count on a number.

Q: What happens when I hit the weekly allowance?

Requests over your plan’s limits return an HTTP 429 error. Personal includes 50M tokens a week and 20 requests per minute. Personal Max raises that to 600M a week and 60 requests per minute for $100/month. Team uses prepaid pay-as-you-go credits with no weekly token cap.

Q: Do I choose between models?

No. You send every request to one model string, pareto. Pareto selects frontier and open models for each request and blends their outputs into one answer.

Q: Is my code kept?

Personal and Personal Max are not our zero data retention tier: prompts and responses may be kept up to 30 days, as section 5(c) of the Terms explains. For work that must not be kept, ask about the Zero Data Retention API on the Private AI page.

Add one step after your tests. When they fail, a headless agent reads the log, fixes the code and reruns the suite. Your script keeps the patch only if the tests pass untouched. The agent runs on Pareto, Unbiased’s model, which selects frontier and open models for each request and blends their outputs into one answer.

Choose Personal · $10/month See the recipe →

Tested live with Codex CLI 0.162.1 and Factory Droid 0.245.050M tokens a week

CI · shop · test

python3 -m unittest 2>&1 | tee ci.log

AssertionError: Lists differ: [4, 5, 6, 7] != [0, 1, 2, 3]
AssertionError: Lists differ: [] != [8, 9]
FAILED (failures=2)

UNBIASED_API_KEY=*** sh .github/autofix-codex.sh

I’ll inspect ci.log and the failing test paths, then patch the application code and rerun the full suite.

exec sed -n '1,240p' ci.log

The failures point to an off-by-one/zero-based page calculation in paginate.

exec cat shop/pages.py tests/test_pages.py

apply_patch Update File: shop/pages.py

- start = page * per_page+ start = (page - 1) * per_page

exec python3 -m unittest

Ran 6 tests in 0.000s
OK

Fixed the one-based page offset in shop/pages.py:5. python3 -m unittest now passes all 6 tests.

python3 -m unittest && git diff --exit-code -- tests

OK · tests unchanged · autofix.patch written

model pareto · provider unbiased47,580 tokens

Patch saved as autofix.patch for review

A replay of autofix-codex.sh from our run, with the JSON events Codex wrote to codex.jsonl shown as steps. Not a screenshot.

Tools that do this

Each of these talks to Pareto with a base URL, an API key and the model string pareto. Pick the one you already use.

The recipe: a config, a script and one workflow step

Both versions do the same thing. Your test step saves its output to ci.log. If it fails, a short script hands the log to the agent, then reruns the tests itself and checks that nothing under tests/ changed. Only then does it write autofix.patch. The build stays red, so a person still reads the patch before anything merges.

Get an Unbiased API key and store it as a secret
Choose a plan, then create a key in the Unbiased platform. New accounts are reviewed and approved by a person before the key works. In GitHub, save it as the repository secret UNBIASED_API_KEY.
Save the test output to a file
Run your suite with 2>&1 | tee ci.log and set shell: bash on that step. GitHub only adds pipefail when the shell is named, and without it the step passes even when the tests fail.
Point the agent at Pareto
Codex reads a custom provider from ~/.codex/config.toml; Droid reads a custom model from ~/.factory/settings.json. Both use base URL https://api.unbiased.ai/v1 and the model name pareto, typed by hand: Unbiased does not list models at /v1/models.
Run the autofix script only when the tests fail
Give the step if: failure() && steps.tests.outcome == 'failure' and pass the key to that step alone. The prompt names the log, says to fix the code and not the tests, and asks the agent to rerun the suite.
Check the fix yourself and keep the patch
Do not take the agent’s word for it. The script reruns python3 -m unittest, fails if git diff --exit-code -- tests finds a change, and saves git diff as an artifact for a person to apply.

Tested on October 20, 2026Codex CLI 0.162.1 and Factory Droid 0.245.0 on Python 3.15.0, model pareto. Both tools read the CI log, changed one line in shop/pages.py and left the tests alone, and all 6 tests passed on a clean rerun, 3 runs out of 3 each. Run record.

OpenAI Codex CLI

codex exec -s workspace-write lets Codex edit files and run commands inside the checkout without asking. --json writes every step to codex.jsonl, which you can keep as a log of what it ran. --full-auto is gone in current releases; use -s workspace-write. < /dev/null stops Codex from waiting for more input on stdin.

Install: npm install -g @openai/codex@0.162.1
Script tested on macOS with Codex CLI 0.162.1, 3 runs out of 3. Not yet tested on Linux
On Linux the sandbox uses bubblewrap. The sysctl line is the same change openai/codex-action makes on GitHub-hosted Ubuntu runners so the sandbox can start
ignore_default_excludes = false keeps variables named like *KEY* out of the commands Codex runs. Without it, our test command could read UNBIASED_API_KEY

~/.codex/config.toml

model = "pareto"
model_provider = "unbiased"

[model_providers.unbiased]
name = "Unbiased"
base_url = "https://api.unbiased.ai/v1"
env_key = "UNBIASED_API_KEY"
wire_api = "responses"

[shell_environment_policy]
ignore_default_excludes = false

.github/autofix-codex.sh

#!/bin/sh
# Runs after the test step fails. ci.log holds the failing output.
set -e
PROMPT="CI failed: python3 -m unittest exited 1 and its output is in ci.log."
PROMPT="$PROMPT Find the cause and fix the code, not the tests."
PROMPT="$PROMPT Then run python3 -m unittest and confirm every test passes."
codex exec -s workspace-write --json "$PROMPT" < /dev/null > codex.jsonl
python3 -m unittest            # the fix has to pass on a clean rerun
git diff --exit-code -- tests  # and must leave the tests alone
git diff > autofix.patch

.github/workflows/test.yml

name: test
on: push
jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
      - name: Run tests
        id: tests
        shell: bash
        run: python3 -m unittest 2>&1 | tee ci.log
      - name: Ask Codex for a fix
        if: failure() && steps.tests.outcome == 'failure'
        env:
          UNBIASED_API_KEY: ${{ secrets.UNBIASED_API_KEY }}
        run: |
          npm install -g @openai/codex@0.162.1
          # Ubuntu 24.04 blocks the user namespaces the Codex sandbox needs
          sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0
          mkdir -p ~/.codex
          cat > ~/.codex/config.toml <<'TOML'
          model = "pareto"
          model_provider = "unbiased"

          [model_providers.unbiased]
          name = "Unbiased"
          base_url = "https://api.unbiased.ai/v1"
          env_key = "UNBIASED_API_KEY"
          wire_api = "responses"

          [shell_environment_policy]
          ignore_default_excludes = false
          TOML
          sh .github/autofix-codex.sh
      - uses: actions/upload-artifact@v4
        if: failure() && steps.tests.outcome == 'failure'
        with:
          name: autofix
          path: autofix.patch

We ran the config and script above on macOS, on a fresh copy of a failing repo with an isolated home folder. We have not run this workflow on GitHub or tested Codex’s sandbox on Linux, so treat the workflow as a starting point: it installs Codex, opens up the user namespaces its sandbox needs, writes the same config and calls the same script.

Factory Droid

droid exec --auto medium may edit files, install trusted packages and run builds and tests, and stops before git push. -o json writes the final result, turn count and token usage to droid.json. Pick the model with custom:Pareto-0: the display name, then its position in customModels starting at 0.

Install: curl -fsSL https://app.factory.ai/cli | sh
Tested with Factory Droid 0.245.0, 3 runs out of 3
In our local runs on macOS, Droid needed no Factory sign-in and no FACTORY_API_KEY. Factory’s docs say to set FACTORY_API_KEY, so the workflow passes it from a repository secret

~/.factory/settings.json

{
  "customModels": [
    {
      "model": "pareto",
      "displayName": "Pareto",
      "baseUrl": "https://api.unbiased.ai/v1",
      "apiKey": "${UNBIASED_API_KEY}",
      "provider": "generic-chat-completion-api",
      "maxOutputTokens": 32000
    }
  ]
}

.github/autofix-droid.sh

#!/bin/sh
# Runs after the test step fails. ci.log holds the failing output.
set -e
PROMPT="CI failed: python3 -m unittest exited 1 and its output is in ci.log."
PROMPT="$PROMPT Find the cause and fix the code, not the tests."
PROMPT="$PROMPT Then run python3 -m unittest and confirm every test passes."
droid exec --auto medium --model "custom:Pareto-0" -o json "$PROMPT" > droid.json
python3 -m unittest            # the fix has to pass on a clean rerun
git diff --exit-code -- tests  # and must leave the tests alone
git diff > autofix.patch

.github/workflows/test.yml

name: test
on: push
jobs:
  test:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
      - name: Run tests
        id: tests
        shell: bash
        run: python3 -m unittest 2>&1 | tee ci.log
      - name: Ask Droid for a fix
        if: failure() && steps.tests.outcome == 'failure'
        env:
          UNBIASED_API_KEY: ${{ secrets.UNBIASED_API_KEY }}
          FACTORY_API_KEY: ${{ secrets.FACTORY_API_KEY }}
        run: |
          curl -fsSL https://app.factory.ai/cli | sh
          export PATH="$HOME/.local/bin:$PATH"
          mkdir -p ~/.factory
          cat > ~/.factory/settings.json <<'JSON'
          {
            "customModels": [
              {
                "model": "pareto",
                "displayName": "Pareto",
                "baseUrl": "https://api.unbiased.ai/v1",
                "apiKey": "${UNBIASED_API_KEY}",
                "provider": "generic-chat-completion-api",
                "maxOutputTokens": 32000
              }
            ]
          }
          JSON
          sh .github/autofix-droid.sh
      - uses: actions/upload-artifact@v4
        if: failure() && steps.tests.outcome == 'failure'
        with:
          name: autofix
          path: autofix.patch

We ran the settings and script above on a fresh copy of a failing repo with an isolated home folder. We have not run the workflow file on GitHub: it installs Droid, writes the same settings and calls the same script. Commands Droid runs can read UNBIASED_API_KEY, so use it on code you trust.

From ci.log before, and from autofix.patch after (same one-line fix in all 6 runs)

FAIL: test_first_page (tests.test_pages.PaginateTest.test_first_page)
AssertionError: Lists differ: [4, 5, 6, 7] != [0, 1, 2, 3]
FAIL: test_last_partial_page (tests.test_pages.PaginateTest.test_last_partial_page)
AssertionError: Lists differ: [] != [8, 9]
Ran 6 tests in 0.001s
FAILED (failures=2)

--- a/shop/pages.py
+++ b/shop/pages.py
@@ -2,7 +2,7 @@ def paginate(items, page, per_page):
     """Return one page of items. Pages are numbered from 1."""
     if page < 1 or per_page < 1:
         raise ValueError("page and per_page must be at least 1")
-    start = page * per_page
+    start = (page - 1) * per_page
     return items[start:start + per_page]

Our test repo is small on purpose: one off-by-one in paginate(), two failing tests out of six. Real failures can need more context, more turns and more tokens, and some need a person.

What it costs on Pareto

An agent fix is many requests, not one: Codex and Droid read the log, open files, patch and rerun the tests, sending the conversation so far each time. In our 6 runs, one fix used between 37,688 and 55,455 tokens as the tools reported them, most of it repeated input the tools marked as cached. Each took 8 to 17 seconds. These figures cover a one-line bug in a six-test repo, not yours.

Personal at $10/monthFor one person: 50M
Personal Max at $100/monthFor one person: 600M

This page shows plan allowances only; it does not turn our run into a per-fix price. An autofix step that runs for a business is production use, which needs Team or a written agreement. Rates are on Pricing. Run record.

One key for every tool on this page

Pick a monthly allowance, or pay per token on Team.

Personal

$10/month

50M tokens a week
20 requests per minute
For one person

Choose Personal

Personal Max

$100/month

600M tokens a week
60 requests per minute
For one person

Choose Personal Max

Team

Pay as you go

Prepaid credits at published rates
300 requests per minute
No weekly token cap

Get started with Team

Subscriptions are for one person and are not unlimited. New API accounts are reviewed by a person before access turns on, and commercial or production use needs a separate written agreement. Rates and limits are on Pricing.

Questions about this recipe

Will it push a fix to my branch?

Not with these files. Both scripts end by writing autofix.patch, the workflow uploads it as an artifact, and the build stays red. A person downloads the patch, reads it and applies it with git apply autofix.patch. Droid at --auto medium is not allowed to git push, and the workflow’s token only has contents: read.

What stops the agent from editing the tests until they pass?

Two things. The prompt says to fix the code and not the tests, and the script does not trust the prompt: git diff --exit-code -- tests exits non-zero if anything under tests/ changed, so no patch is written. In all 6 of our runs the only changed file was shop/pages.py.

Can the commands the agent runs see my API key?

With Codex, not if you keep ignore_default_excludes = false: in our check, a command Codex ran could not see UNBIASED_API_KEY with that line and could see it without it. With Droid, it could. Either way your tests and their dependencies run in the same job, so pass the key only to the autofix step and run this on your own code, not on untrusted pull requests.

Can I use openai/codex-action instead of installing Codex myself?

The action’s inputs include responses-api-endpoint and model, so it may be pointed at https://api.unbiased.ai/v1/responses with model pareto. We have not tested that. The version on this page installs the CLI and uses the same config.toml we ran.

Codex prints that model metadata for pareto was not found. Is the run broken?

No. Codex has metadata for OpenAI’s own model names and uses fallback metadata for any other name. It printed that warning in every one of our runs, which then fixed the bug and passed the rerun. Format compatibility is not a promise that every Codex or Droid feature works, so tell us if one doesn’t.

How many fixes fit in a plan?

Personal includes 50M tokens a week at 20 requests a minute. Our fixes used about 38,000 to 56,000 tokens each, but a bigger repo or a harder failure can use several times that, so measure a few of your own before you count on a number.

What happens when I hit the weekly allowance?

Requests over your plan’s limits return an HTTP 429 error. Personal includes 50M tokens a week and 20 requests per minute. Personal Max raises that to 600M a week and 60 requests per minute for $100/month. Team uses prepaid pay-as-you-go credits with no weekly token cap.

Do I choose between models?

No. You send every request to one model string, pareto. Pareto selects frontier and open models for each request and blends their outputs into one answer.

Is my code kept?

Personal and Personal Max are not our zero data retention tier: prompts and responses may be kept up to 30 days, as section 5(c) of the Terms explains. For work that must not be kept, ask about the Zero Data Retention API on the Private AI page.

Same key, same model string, different job. See every use case. Or find your tool on Works with.