Personal
$10/month
- 50M tokens a week
- 20 requests per minute
- For one person
Test Pareto on your own prompts before you switch
Public benchmarks are not your workload. Put the prompt you run today in one file, your real inputs and the answers you expect in a CSV, and grade Pareto on them next to the model you use now. Pareto is Unbiased’s model: it selects frontier and open models for each request and blends their outputs into one answer. Your test cases decide whether that answer is good enough.
Tested live with promptfoo 0.124.1 and openai 3.35.150M tokens a week
Tools that do this
Each of these talks to Pareto with a base URL, an API key and the model string pareto. Pick the one you already use.
The recipe: one prompt, one CSV, two ways to run it
Both versions read the same prompt.txt and cases.csv, send each case to Pareto and mark it PASS or FAIL. Add the model you use today where the files say so, and every case runs against both. Pick promptfoo for a results matrix, repeat runs and many assertion types; pick the script if you want to read every line that runs.
Get an Unbiased API key
Choose a plan, then create a key in the Unbiased platform. New accounts are reviewed and approved by a person before the key works.
Save the prompt you run today
Put it in
prompt.txtexactly as your app sends it, with{{ message }}where the input goes.Write test cases from real traffic
One row per input in
cases.csv. Every column becomes a variable; the__expectedcolumn holds the check, for exampleicontains:billing. Copy real messages from your logs, including the awkward ones.List Pareto next to your current model
Provider id
openai:chat:pareto,apiBaseUrl: https://api.unbiased.ai/v1andapiKeyEnvar: UNBIASED_API_KEY. Typeparetoby hand: Unbiased does not list models at/v1/models. Uncomment the current-model entry and give it the provider id you run today.Run it and read the failures
Run
npx promptfoo eval, thennpx promptfoo viewfor the side-by-side matrix. For each FAIL, decide whether the answer is wrong or the test is, then fix the prompt or the case and run it again.
✓Tested on October 20, 2026promptfoo 0.124.1 (Node 26.11.1) and openai (Python) 3.35.1 on Python 3.15.0, model
pareto. promptfoo graded four support-triage cases against Pareto and all four passed in 3 runs out of 3; the plain OpenAI SDK script got the same four labels in 3 runs out of 3. Only Pareto was run, so there is no comparison result. Run record.promptfoo
Each provider in providers gets a column, each row of cases.csv gets a row, and every cell is graded by the __expected check plus the one-word rule in defaultTest.
- Runs with
npx, no install step; needs Node.js - Tested with promptfoo 0.124.1 on Node.js 26.11.1
- Set
PROMPTFOO_DISABLE_TELEMETRY=1to turn off promptfoo’s telemetry
promptfooconfig.yaml
description: Support triage, Pareto next to my current model
prompts:
- file://prompt.txt
providers:
- id: openai:chat:pareto
label: Pareto
config:
apiBaseUrl: https://api.unbiased.ai/v1
apiKeyEnvar: UNBIASED_API_KEY
max_tokens: 4096
# Your current model: uncomment and use the provider id you run today.
# - id: openai:chat:your-current-model
# label: Current
defaultTest:
assert:
- type: javascript
value: output.trim().split(/\s+/).length === 1
tests: file://cases.csvprompt.txt
Classify this support message as exactly one word:
billing (charges, refunds, invoices), shipping, bug or other (anything else, including questions about prices).
Message: {{ message }}cases.csv
message,__expected "Charged twice for my annual plan ($96 each). pls refund one.",icontains:billing "order #A-10442 never showed up, tracking says delivered??",icontains:shipping "app crashes every time i export a pdf on ipad",icontains:bug "do you have a student discount?",icontains:other
Shell
export UNBIASED_API_KEY=your-unbiased-api-key
npx promptfoo eval
# open the side-by-side results matrix in your browser
npx promptfoo viewYour current model keeps its own key: an openai:chat: model reads OPENAI_API_KEY, and other providers read theirs. If it sits behind another OpenAI-compatible endpoint, give that entry its own apiBaseUrl and apiKeyEnvar.
OpenAI Python SDK
compare.py sends every case in cases.csv to every model in MODELS and prints a verdict, the answer, the token count the API reported and the time taken. Same prompt.txt and cases.csv as the promptfoo tab.
- Install:
pip install openai - Tested with openai 3.35.1 on Python 3.15.0
OpenAI()with no arguments readsOPENAI_API_KEY; passbase_urlandapi_keyfor any other OpenAI-compatible endpoint
compare.py
import csv
import os
import time
from openai import OpenAI
MODELS = [
('Pareto', 'pareto', OpenAI(
base_url='https://api.unbiased.ai/v1',
api_key=os.environ['UNBIASED_API_KEY'],
)),
# Your current model: uncomment and fill in its name and client.
# ('Current', 'your-current-model', OpenAI()),
]
prompt = open('prompt.txt').read()
for row in csv.DictReader(open('cases.csv')):
want = row['__expected'].split(':', 1)[1]
for label, model, client in MODELS:
start = time.time()
reply = client.chat.completions.create(
model=model,
messages=[{'role': 'user', 'content': prompt.replace('{{ message }}', row['message'])}],
)
answer = reply.choices[0].message.content.strip()
verdict = 'PASS' if want in answer.lower() else 'FAIL'
print(f"{verdict} {label} {answer!r} {reply.usage.total_tokens} tokens {time.time() - start:.1f}s")Shell
pip install openai export UNBIASED_API_KEY=your-unbiased-api-key python compare.py
The script only checks that the expected label appears in the answer. For anything richer, such as JSON checks, model-graded rubrics or repeat runs, use promptfoo.
Output from our runs (Pareto only)
# npx promptfoo eval, end of output Total Tokens: 288 Provider: 288 (278 prompt, 10 completion) Results: ✓ 4 passed (100%) 0 failed (0%) 0 errors (0%) # python compare.py PASS Pareto 'billing' 126 tokens 2.8s PASS Pareto 'shipping' 108 tokens 1.7s PASS Pareto 'bug' 114 tokens 1.5s PASS Pareto 'other' 129 tokens 1.6s
We ran Pareto only, so this page shows no comparison with another model. The side-by-side result is the one you get from your own cases.
What an eval costs on Pareto
Each test case is one request per provider, so Pareto’s column uses Pareto tokens and your current model’s column bills to that provider. Short classification prompts like ours are cheap; long documents and long answers cost more.
- Personal at $10/monthFor one person
- 50M tokens a week, 20 requests a minute
- Personal Max at $100/monthFor one person
- 600M tokens a week, 60 requests a minute
In our runs each test case used 67 to 76 tokens through promptfoo, 288 for all four cases, and 70 to 147 through the script, as the API reported. promptfoo sends temperature: 0 and an output limit by default; the script sends neither. Your prompts will differ, so run a small set first and read the token totals promptfoo prints. Plan selection does not itself authorize production use. Confirm commercial or production use in a separate written agreement with Unbiased before moving live traffic. Rates and limits are on Pricing. Run record.
One key for every tool on this page
Pick a monthly allowance, or pay per token on Team.
Personal Max
$100/month
- 600M tokens a week
- 60 requests per minute
- For one person
Team
Pay as you go
- Prepaid credits at published rates
- 300 requests per minute
- No weekly token cap
Subscriptions are for one person and are not unlimited. New API accounts are reviewed by a person before access turns on, and commercial or production use needs a separate written agreement. Rates and limits are on Pricing.
Questions about this recipe
What did the eval catch in your own test?
Our first run used a shorter prompt that listed the four labels without defining them. Pareto answered billing for “do you have a student discount?”, which our test expected as other, so 3 of 4 cases passed. Either answer is defensible, which is the point: the eval showed the prompt was ambiguous. We added a definition of each label, the prompt shown above, and every case passed in all six runs after that.
How do I add the model I use today?
In promptfoo, uncomment the Current entry and replace openai:chat:your-current-model with the provider id you run now; promptfoo supports many providers besides OpenAI. In compare.py, uncomment the second MODELS line and set the model name and client. Each model keeps its own API key.
Will Pareto give the same answer every run?
In our runs it did: the same four labels in all three promptfoo runs and all three script runs. Answers to judgment calls can still move between runs. Add --repeat 3 --no-cache to npx promptfoo eval to run each case several times and see how stable your own cases are. Keep --no-cache: without it, promptfoo reuses cached answers for each repeat.
Where do my prompts and results go?
Each test case is sent to the providers you list. promptfoo keeps eval history on your machine in ~/.promptfoo (change it with PROMPTFOO_CONFIG_DIR) and turns off its telemetry with PROMPTFOO_DISABLE_TELEMETRY=1. Use -o results.json to save a run to a file.
Can promptfoo grade with Pareto too?
Yes, for model-graded assertions such as llm-rubric. They do not inherit the providers list, so set defaultTest.options.provider to the same openai:chat:pareto entry with its config. We did not run a model-graded check for this page.
What happens when I hit the weekly allowance?
Requests over your plan’s limits return an HTTP 429 error. Personal includes 50M tokens a week and 20 requests per minute. Personal Max raises that to 600M a week and 60 requests per minute for $100/month. Team uses prepaid pay-as-you-go credits with no weekly token cap.
Do I choose between models?
No. You send every request to one model string, pareto. Pareto selects frontier and open models for each request and blends their outputs into one answer.
Is my code kept?
Personal and Personal Max are not our zero data retention tier: prompts and responses may be kept up to 30 days, as section 5(c) of the Terms explains. For work that must not be kept, ask about the Zero Data Retention API on the Private AI page.
More jobs Pareto does
Same key, same model string, different job. See every use case. Or find your tool on Works with.
Product names belong to their owners. Unbiased is not affiliated with or endorsed by the makers of the tools on this page.
Let your own test cases decide
One prompt file, one CSV and an API key. Pareto starts at $10 a month for 50M tokens a week.