# Pairwise human judgments

Origin: lmsys/mt_bench_human_judgments, human split, revision f7d2896d2cc5d80f8b55c2bbc722613555233c25.
Source: https://huggingface.co/datasets/lmsys/mt_bench_human_judgments
License checked in the pinned dataset card: CC BY 4.0.
License URL: https://creativecommons.org/licenses/by/4.0/
Attribution: Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (2023), https://arxiv.org/abs/2306.05685.

We selected 100 test pairs and 20 dev pairs, turn one only, with non-equal human preferences and two annotators per pair. Questions are disjoint between splits. Sampling seed: 20261027. Round-robin sampling covers writing, roleplay, reasoning, math, coding, extraction, STEM and humanities. The first annotator in lexical identifier order provides gold; the second provides a human-to-human reference. Selection does not require agreement. Labels are existing human annotations, not labels generated by an AI. No new human labeling is claimed.

Model identities are blinded to A/B. Original candidates are historical models from the source study; none is Pareto. Generator models: source dataset models, blinded to A/B. Every alternate item reverses the original display positions and labels together. Source question ids, annotator ids and original pair hashes are in labels.jsonl. Em dash typography becomes spaced hyphens; answers otherwise retain their source text. Pairs containing site-prohibited terms or pictographs and empty answers were excluded before sampling. This filtering narrows representativeness.

Public benchmark contamination is possible. This older conversational set does not establish release-gating quality on current models, private tasks, or production traffic. Human spot-check of the selected examples and the typography normalization is still requested.

The prompt is frozen without tuning or scored-test inspection. The headline is Cohen's kappa against the first human label, using the original display order and majority prediction across five repeats. Equal preferences and malformed calls are explicit prediction classes, never dropped. Swapped-order consistency and raw agreement are secondary. Both orders are called separately, with usage summed per pair. Different explanations are output variation, not proof of statistical independence.

Deviations: the separate HelpSteer rubric set and promptfoo smoke integration remain follow-ups, not tested claims. Human-to-human reference uses the actual selected pair labels rather than a paper's figure. Fleiss and Krippendorff repeat agreement will be reported from the retained raw decisions; no stability claim is inferred from response ids.

Data: CC BY 4.0. Recipe, checker and runner code: MIT, copyright Unbiased 2026. See LICENSE.txt.