# Source and labeling

This is a synthetic, fictional source-relative verification set, created by a deterministic Python generator authored with Pareto on 2026-10-27. No external model was called to draft it. Source passages describe fictional observatories; they are not claims about actual people or institutions. The generator is scripts/evals/verify-claims-against-sources/generate_fixture.py in the authoring repository.

## License

Unbiased dedicates the original source passages, draft text and labels to CC0 1.0 Universal: https://creativecommons.org/publicdomain/zero/1.0/. No third-party text is included, so no Wikipedia, government-text or FEVER attribution is required. Code is MIT; see LICENSE.txt. This license applies only to this synthetic fixture, not to third-party tools.

## Construction and splits

Labels are fixed before text is generated. Each draft has eight supported claims, one contradicted claim and one claim not in the supplied sources. The order is rotated independently of the label list. Test: 10 drafts, 100 pre-split claims, including 20 seeded errors (10 contradicted and 10 not in sources). Dev: three separate drafts, 30 claims. Each request contains the complete source packet and draft, but asks about one identified claim. Label fields are never included in requests.

Labels set by the generator, spot-checked by us: automated construction and checker assertions plus an agent review of the source/claim pairs. This is NOT a completed human spot-check or two-human adjudication. Human review is pending. No human agreement statistic is claimed.

## Scope and limitations

This replaces the card's Sonnet-authored government/Wikipedia drafts to avoid unapproved third-party inference spend and to use generator-intended labels. There is no Sonnet self-check row. Pareto checks template-generated prose, not drafts produced by a separate model. Pareto's model selection may include a drafting model in real use; this fixture has no external drafting model.

The sources are short, the errors are deliberately planted, and templates overlap between dev and test. This is an easy controlled test, not evidence of open-web verification, realistic draft quality or long-source accuracy. The metric is verify-only: claims are already split and given ids. End-to-end claim extraction is not measured. Claims within a draft share context; a later claim-level paired interval is not a draft-clustered generalization interval.

Headline: seeded-error catch rate (recall). Either contradicted or not_in_sources counts as flagging a seeded error. Secondary: caught-error count, false-alarm count/rate over supported claims, exact quote substring precision over nonempty quotes, label accuracy and invalid-output rate. Empty quotes do not count as matched quotations. Invalid outputs count as misses on seeded errors and are reported separately. Rates are fractions, not percentages. Repeated counts use per-claim medians; recall uses summed counts.

Prompt frozen before any inference. No test-split tuning. The initial full run completed but its receipt was rejected: the count-based headline incorrectly included an empty classification-label array. That optional array was removed, the corrected metadata was committed, and a replacement full run was required. The prompt, fixture, recipe and checker did not change; the rejected run was not selected against the replacement by score. Baseline spend is not approved. Repeated output variation does not establish statistical independence, and a ceiling result is anticipated on this small synthetic set.