# text-to-sql fixture provenance

## What this is

50 test and 15 dev plain-English questions over a SQLite store with the
Chinook schema (11 tables) and fresh synthetic seed data written for this
evaluation. Gold SQL is fixed per question; a model answer is correct when
it executes read-only on two different seeded databases and returns the
same multiset of rows as the gold query (ordered comparison only when the
gold query has ORDER BY).

## Origin and license

- Schema: Chinook Database by Luis Rocha, https://github.com/lerocha/chinook-database,
  MIT license (copy at LICENSE.txt). Version 1.4.5 `Chinook_Sqlite.sql`,
  fetched 2026-10-27. Only the CREATE TABLE statements are shipped
  (`schema.sql`); no Chinook rows are included.
- Data (`seed.sql`): generated for this evaluation by a deterministic
  Python script (`scripts/evals/text-to-sql/generate_fixture.py`, MIT,
  seeded 2026-10-27). Synthetic artists, albums, tracks, customers,
  invoices and playlists. No real personal data. The dataset as a whole
  is released CC0 (`LICENSE.txt`); Chinook's MIT notice is preserved here.
- The plan's optional BIRD mini-dev subset is not included.

## Labeling method

Generator-intended labels: the authoring agent wrote each gold SQL query
alongside its question in a fixed generator table. This is intended SQL
semantics, not independent human gold and not proof of natural-language
correctness. Automated self-checks execute the gold queries; they cannot
establish that the question means what the SQL does.

This replaces the card's two-human gold-SQL process for now: labels are
set by the generator, spot-checked by us with automated checks (every
gold query executes on both seeded databases, and the full fixture
passes the repository lint). Questions test-46 and test-48 have empty
gold results on the original seed; the changed seed must also match its
own gold result. Empty outputs can make some incorrect queries pass. A human
spot-check of the 50 test questions is marked as pending in the chunk
status; the page copy discloses the same.

## Generator models

- Question authoring and code: a Pareto session (Claude Opus 5.5 agent on
  2026-10-27) wrote the generator, the gold SQL intent and the question
  text. No GPT-6.1 Sol or other scored-arm model produced any fixture.
- No other model was used.

## Contamination note

Chinook appears in many public text-to-SQL tutorials, so models may have
seen similar questions. We mitigate but do not eliminate this: the seed
data is synthetic and unpublished, answers depend on this data, and
questions avoid the canonical tutorial phrasings. Contamination would
affect the arms differently; no equal effect is assumed. This synthetic
set does not establish accuracy on unfamiliar schemas or real workloads.

## Safety

The checker only ever executes SQL inside the macOS eval sandbox
(`scripts/evals/lib/sandbox.py`), against a read-only SQLite connection
with an authorizer that denies writes, ATTACH, PRAGMA, extensions and
non-SELECT statements, plus row, statement-length and step limits.