In blind verification, model selection matters more than source type: Bradley-Terry ranking interleaves human coders and LLMs, with per-model win rates spanning significant human advantage to numerical LLM advantage¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Per-model win rates against human coders ranged from 58.8% (Claude Opus, p = 0.092) down to 37.3% (Claude Haiku, p = 0.013); GPT-4o (39.8%, p = 0.048) and Claude Haiku were preferred significantly less often than human coders. → Liu 2026 q2 i? The Bradley-Terry ranking interleaves the eight sources: Claude Opus ranks first but is statistically indistinguishable from Coder 1, GPT-5.5 and Gemini Flash rank above two of the three human coders, and the 0.875 log-unit spread among LLMs dwarfs the 0.063 log-unit gap between the best LLM and the best human coder. → Liu 2026 (2)

Evidence¶

Liu 2026¶

Liu, A., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., He, K., & Sun, M. (2026). Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth. Preprint, no identifier printed in text. https://arxiv.org/abs/2607.28890 doi:10.48550/arxiv.2607.28890

q2 · i? · design · r2

Per-model binomial tests of LLM win rates in decisive Human vs. LLM pairs of the blind verification protocol (Table 4; roughly 102–106 decisive pairs per model). Win rates ran from 58.8% (Claude Opus) to 37.3% (Claude Haiku); only GPT-4o and Claude Haiku fell significantly below 50%.

"Claude Opus and Gemini Flash were preferred over human coders more often than not, though neither difference reaches per-model significance. GPT-5.5 shows no directional preference. GPT-4o and Claude Haiku were preferred significantly less often than human coders (p = 0.048 and p = 0.013), confirming lower coding quality as judged by the verifier."

Liu 2026 (2)¶

Liu, A., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., He, K., & Sun, M. (2026). Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth. Preprint, no identifier printed in text. https://arxiv.org/abs/2607.28890. doi:10.48550/arxiv.2607.28890

q2 · i? · design · r2

Bradley-Terry model fit to all 801 decisive blind comparisons over the eight verified sources (Figure 3, log-scale ability estimates with 95% bootstrap intervals). The figure note adds that the "0.875 log-unit spread among LLMs dwarfs the 0.063 log-unit gap between the best LLM and the best human coder".

"Claude Opus occupies the top position with a confidence interval overlapping substantially with Coder 1, so the two are statistically indistinguishable. GPT-5.5 and Gemini Flash rank above two of the three human coders, Claude Haiku ranks below all of them, and GPT-4o is the only source whose interval does not overlap with any human coder."

Discussion¶

Related Claims¶

• Human-LLM agreement is moderated by code properties, multi-model consensus, and model-reported confidence — related
• Agreement metrics and expert quality judgments diverge in both directions: human consensus can encode shared conservative bias that the verifier rejects in favor of LLM coding — related
• Blind expert verification shows no overall preference for human over LLM qualitative coding when sources are judged symmetrically — a broader claim this one bears on
• On codes requiring recognition of pedagogical intent not explicitly stated, LLMs over-apply and the blind verifier rejects their application in favor of human coding — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• Human-human agreement (average κ = 0.644) was lower than the best LLM-LLM agreement (average κ = 0.856) across constructs — related
• Zero-shot prompt type has minimal impact on LLM-human coding concordance — related
• Confidence-aware selective test-time scoring achieves the best average agreement with expert rubric scoring across six NGSS drawing items — related
• CLARA annotations align with blinded educator judgments, with 0.81 pairwise agreement and highest-rated educational interpretability — related
• CLARA achieves the highest agreement with blinded human developmental ranking judgments compared with readability and prompting baselines — related
• In blind A/B comparison, raters prefer personalised (Adaptive) questions over non-personalised (Random-topic) questions about 68–69% of the time in Biology and Chemistry — related
• Human raters' preferences align with the LLM judge on TutorBench (Pearson r = 0.82 across metric-level win rates) — related
• LLM-judge evaluation of tutoring sycophancy shows systematic self-judge blind spots and missed sycophancy even under judge consensus — related
• The cross-family LLM judge panel agrees with human domain experts on every judged axis, reaching the expert ceiling on answer-holding — related
• The ConvTutor–PedTutor helpfulness ordering is judge-contingent, reversing between the Opus and Sol judges on two of three tutor bases — a narrower finding that bears on this claim
• Automated metrics and LLM judges can misalign with expert human preferences in non-verifiable educational evaluation — related
• LLM graders show model-specific precision/recall trade-offs, so a single metric is insufficient to judge grading quality — related
• Students engaged deeply with LLM feedback, with coded evidence of reflection, rebuttal, and acceptance in the large majority of responses — related
• The LLM judge agrees with human annotators at 87.7% (κ = 0.842) on consensus pairs, but severity-rating agreement is uneven and medium-tier boundaries are uncertain
• GPT-4.1-Mini with back-translation matches the best LLM-judge (GPT-5) at 10.3x lower evaluation cost — related
• An automated LLM assessor rates 3E clinical communication skills with agreement comparable to individual human raters — related
• No significant differences emerged between GPT-3.5 Turbo and Claude 3 Opus on affective, collaborative, or performance measures — related
• LLM coders replicate many human discursive moves but lack questioning, adaptive responsiveness, and emotional expressiveness — related
• Hierarchical Acceptance Rate aligns with expert acceptance decisions better than single-dimension and holistic LLM judges — related