unbiased ai

Put Pareto through your promptfoo evals

Before a new model touches your prompts, make it sit your exam. Add Pareto to promptfooconfig.yaml as an OpenAI-compatible provider, list it next to the model you use now, and promptfoo eval runs every test case against both. Pareto selects frontier and open models for each request and blends their outputs into one answer. Your assertions decide whether that answer passes.

Choose Personal · $10/month See the setup →

Tested live with promptfoo 0.124.150M tokens a week

An illustration of promptfoo on Pareto. Not a screenshot.

Set it up in promptfooconfig.yaml

promptfoo talks to OpenAI-compatible APIs through its OpenAI provider. Give it Unbiased’s base URL, the name of the environment variable that holds your key, and the model name pareto. These are the settings from our live run.

  1. Get an Unbiased API key

    Choose a plan, then create a key in the Unbiased platform. New accounts are reviewed and approved by a person before the key works.

  2. Add Pareto as a provider

    Use the id openai:chat:pareto. The chat: part tells promptfoo to call Chat Completions, which promptfoo’s docs recommend spelling out for a compatible gateway.

  3. Point it at Unbiased

    Set apiBaseUrl to https://api.unbiased.ai/v1. Keep /v1 and leave off /chat/completions: promptfoo adds the endpoint path itself.

  4. Name the key variable

    Set apiKeyEnvar: UNBIASED_API_KEY and export that variable with your key. Unbiased does not list models, so type pareto by hand.

  5. Run the eval

    Run npx promptfoo eval, then npx promptfoo view to see every output with its pass or fail mark.

Tested on October 20, 2026promptfoo 0.124.1 on Pareto (pareto). It ran a three-case eval against Pareto (arithmetic, a capital city and a string reversal) and all three assertions passed. Run record.
promptfooconfig.yaml
prompts:
  - "Answer with only the answer, no other words. {{ question }}"
providers:
  - id: openai:chat:pareto
    label: Pareto
    config:
      apiBaseUrl: https://api.unbiased.ai/v1
      apiKeyEnvar: UNBIASED_API_KEY
      max_tokens: 4096
tests:
  - vars: { question: "What is 17 + 25?" }
    assert: [{ type: contains, value: "42" }]
Terminal
export UNBIASED_API_KEY=your-unbiased-api-key
npx promptfoo eval
# open the results matrix in your browser
npx promptfoo view
Optional: add your current model alongside
providers:
  - id: openai:chat:pareto
    label: Pareto
    config:
      apiBaseUrl: https://api.unbiased.ai/v1
      apiKeyEnvar: UNBIASED_API_KEY
      max_tokens: 4096
  # your current model, with its own key
  - id: openai:chat:gpt-4.1-mini
    label: Current

With --verbose, promptfoo notes Using unknown chat model: pareto. That is expected, and the eval runs normally. Each test case runs once per provider, so a side-by-side eval uses Pareto tokens only for Pareto’s column.

Your tests stay. The model under test changes.

Pareto is one more provider entry. The prompts, test cases and assertions you already wrote run against it unchanged, so you can see how a blend of frontier and open models handles your own cases before you switch.

One model string, several models behind it

Pareto is Unbiased’s blended AI model. On each request it selects frontier and open models and blends their outputs into one answer, so promptfoo talks to one model string, pareto, and never has to pick.

“When the frontier improves, Pareto improves; when frontier prices fall, you should benefit.” From the pricing page.

How a Pareto answer gets made →
An illustration, not a wiring diagram. The blend includes frontier and open models, and its composition can change.

What an agent task costs on Pareto

Agents make many model calls per task, so the price of a finished task matters more than the price of a token. The model card publishes both for Pareto on two public agent benchmarks.

Terminal-Bench 4.0Tasks an agent completes in a terminal
$0.48/task
50.8% solved
DeepSWE v1.1Changes an agent makes in a real codebase
$0.24/task
69.9% solved

Pareto 26.10, preliminary October 1, 2026 runs that may change; denominators and the cost method still require confirmation. Cost is the reported mean USD per task at API rates, not an estimate for your workload. Model card and method.

Pay for promptfoo by the month

One API key works across promptfoo and any other tool you point at Pareto. Pick a monthly allowance, or pay per token on Team.

Personal

$10/month

  • 50M tokens a week
  • 20 requests per minute
  • For one person
Choose Personal

Personal Max

$100/month

  • 600M tokens a week
  • 60 requests per minute
  • For one person
Choose Personal Max

Team

Pay as you go

  • Prepaid credits at published rates
  • 300 requests per minute
  • No weekly token cap
Get started with Team

Subscriptions are for one person and are not unlimited. New API accounts are reviewed by a person before access turns on, and commercial or production use needs a separate written agreement. Rates and limits are on Pricing.

promptfoo on Pareto, questions answered

Why does promptfoo cut Pareto’s answers short?

For non-reasoning requests, promptfoo sends temperature: 0 and a 1,024-token output limit unless you say otherwise. Set max_tokens in the provider’s config (our run used 4096), or set omitDefaults: true to drop both defaults. promptfoo’s docs suggest setting the limit explicitly whenever you compare models.

Do I still need an OpenAI API key?

Not for Pareto. With apiKeyEnvar, promptfoo reads your Unbiased key from the variable you name and does not fall back to OPENAI_API_KEY. You only need an OpenAI key if you also list an OpenAI model to compare against. You can instead set OPENAI_BASE_URL and OPENAI_API_KEY in the environment, but then every openai: provider goes to Unbiased.

Will llm-rubric grade with Pareto?

Only if you ask. Model-graded assertions do not inherit the providers list. To make Pareto the grader, set defaultTest.options.provider to the same openai:chat:pareto entry, with its config.

Why is the cost column empty for Pareto?

promptfoo estimates cost from rates it knows, and pareto is not on its list. Set inputCost and outputCost in dollars per token if you want an estimate. The figure only changes promptfoo’s report; your Unbiased plan decides what you pay.

Can I run these evals in CI?

promptfoo runs the same way in a CI job as on your laptop, so store UNBIASED_API_KEY as a secret and call npx promptfoo eval. Subscriptions are for one person; for a shared or production pipeline, talk to us about a written agreement first. More for builders on the agent builders page.

Is Pareto made by the promptfoo team?

No. Pareto is Unbiased’s model. promptfoo is made by promptfoo (now part of OpenAI), and Unbiased is not affiliated with or endorsed by them. You keep promptfoo and point it at the Unbiased API.

What happens when I hit the weekly allowance?

Requests over your plan’s limits return an HTTP 429 error. Personal includes 50M tokens a week and 20 requests per minute. Personal Max raises that to 600M a week and 60 requests per minute for $100/month. Team uses prepaid pay-as-you-go credits with no weekly token cap.

Do I choose between models?

No. You send every request to one model string, pareto. Pareto selects frontier and open models for each request and blends their outputs into one answer.

Is my code kept?

Personal and Personal Max are not our zero data retention tier: prompts and responses may be kept up to 30 days, as section 5(c) of the Terms explains. For work that must not be kept, ask about the Zero Data Retention API on the Private AI page.

Pareto works in other tools too

The same key and the same model string go into each of these. See them all.

Product names belong to their owners. Except where a guide says otherwise, Unbiased is not affiliated with or endorsed by the makers of the tools on this page.

Make Pareto sit your test suite

One provider entry, one key, the assertions you already trust. Pareto starts at $10 a month for 50M tokens a week.

Choose Personal · $10/month See the setup →