# invoice-data-extraction fixture source

## Origin

Synthetic invoice PDFs generated for this evaluation with
`scripts/evals/invoice-data-extraction/generate_fixture.py` (MIT). Each of the
50 documents is rendered to a digital PDF with reportlab 5.1.0 from fixed
field values, then converted to text with markitdown 0.1.9. No OCR is
involved: these are digital PDFs we generated, not scans.

## Mix

- 40 test records plus 10 dev records, split by `split`.
- 8 page layouts (rotated metadata, summary-first, reversed order, `key = value` form, indented, wide leading, separated sections).
- 4 currencies: USD, EUR, GBP, CAD.
- 2 credit notes with negative amounts (dev-07, test-07).
- 6 planted total errors in the test split (test-02, test-08, test-13, test-21, test-29, test-37) and 2 in dev (dev-02, dev-08): the printed total differs from subtotal plus tax by 7.25.
- 10 records with no purchase order (every fifth), so `purchase_order` is null.
- Vendors are Faker 40.43.0 company names; emails use reserved `example.invalid` domains. No real person or company data.

## License

CC BY 4.0. Generated text and PDFs contain no third-party content.

## Labeling method

Generator-intended labels: every gold value was fixed before PDF rendering.
Automated checker tests and agent inspection of converted dev text are the
checks performed so far, not independent human labeling. Human spot-check is
pending: verify all planted-error invoices and both credit notes against the
PDFs. Intended generator values are not proof that PDF conversion preserved
all content. Regenerate PDFs with the committed generator and pinned versions;
the original local PDFs are in `/tmp/invoice-digital-pdfs/`.

## Generator models

None. No LLM generated any invoice text; the generator is deterministic
Python with a seeded Faker (seed 271027).

## Notes

- Three scored-test attempts were aborted during contract/checker corrections:
  17 responses / 21,290 reported tokens, 27 / 36,131, and 24 / 31,758.
  These were test exposures, not dev tuning, and invalidate a claim that this
  was an untouched holdout. Their 89,179 retained tokens are excluded from the
  final receipt; interrupted requests may have incurred additional usage.
- The final run follows freeze 749cd69135082aa6eab34be26d5a42c1be3d2569.
  Earlier attempts did not produce published receipts. Final outputs were not
  used to revise the frozen prompt, recipe, checker or configuration.
- SDK dev checks and Pydantic AI smoke calls were additional Pareto usage;
  their tokens were not retained. Pydantic smoke covered dev-00, dev-07 and
  dev-02 three times each. An earlier successful response hit a logging error.
- The headline averages twelve normalized field matches per invoice, then
  takes each invoice's median across repeats. The resampling unit is invoice,
  not the field requested by the build card. Line matching ignores row order.
- Planted-error counts come from local subtotal-plus-tax arithmetic, not the
  model's note. Credit-note line signs and tax law are not checked. A missing
  or unparsable total is not counted as a false flag; schema/field metrics must
  be read alongside that count.
- The frozen planning paired_difference_sd=0.08 was an unsupported assumption,
  not a measured paired dev statistic. Disregard the resulting MDE estimate.
  No baseline was run. Budget/token planning remains separate from that MDE.
- Eight layouts rearrange a small, repeated vocabulary. Perfect performance
  on this easy synthetic set does not establish production invoice accuracy.
- Generator reproduction: Python 3.14.6, Faker 40.43.0, reportlab 5.1.0,
  markitdown[pdf] 0.1.9. Run the committed generate_fixture.py in a separate
  checkout; it overwrites fixture/labels and recreates PDFs under /tmp.
  Do not regenerate over a frozen evaluation used for baseline merging.