Blind expert verification shows no overall preference for human over LLM qualitative coding when sources are judged symmetrically¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q2 · i0 negligible
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i0 Among decisive Human vs. LLM comparisons, an independent blind domain expert preferred human coding 51.5% and LLM coding 48.5% of the time, a statistically negligible deviation from chance. → Liu 2026 q2 i? Verification checks passed: no position bias was detected, and Human vs. Human pairs split nearly evenly across the three coders, confirming the task admits meaningful discrimination without favoring any coder. → Liu 2026 (2)

Evidence¶

Liu 2026¶

Liu, A., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., He, K., & Sun, M. (2026). Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth. Preprint, no identifier printed in text. https://arxiv.org/abs/2607.28890 doi:10.48550/arxiv.2607.28890

q2 · i0 · associational · r2

Blind verification protocol: an expert with a doctorate in education, not involved in the original coding and unaware of source identity, judged 855 anonymized pairwise comparisons with position randomized. Of 855 comparisons, "the verifier expressed a decisive preference in 801 cases (93.7%)"; the human-preference effect versus 0.50 was "a negligible effect" (Cohen’s h = 0.029).

"Among the 555 Human vs. LLM pairs, 515 received decisive preferences, with human coding preferred in 265 cases (51.5%) and LLM coding in 250 (48.5%). A binomial test yields p = 0.537, the 95% confidence interval for the human preference rate is [0.470, 0.559], and Cohen’s h relative to 0.50 is 0.029, a negligible effect."

Liu 2026 (2)¶

Liu, A., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., He, K., & Sun, M. (2026). Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth. Preprint, no identifier printed in text. https://arxiv.org/abs/2607.28890. doi:10.48550/arxiv.2607.28890

q2 · i? · associational · r2

Baseline checks within the same blind verification protocol. Position of the two code sets was randomized and showed "No significant position bias" (p = 0.230), and Human vs. Human baseline pairs split nearly evenly across coders, so the aggregate null is not an artifact of presentation order or verifier favoritism.

"No significant position bias was detected (Option A 47.8% vs. Option B 52.2% of decisive cases, p = 0.230). Among Human vs. Human pairs, decisive preferences split nearly evenly across the three coders (53.6%, 48.2%, 48.1%), confirming that the task admits meaningful discrimination without favoring any individual coder."

Discussion¶

Related Claims¶

• Agreement metrics and expert quality judgments diverge in both directions: human consensus can encode shared conservative bias that the verifier rejects in favor of LLM coding — related
• In blind verification, model selection matters more than source type: Bradley-Terry ranking interleaves human coders and LLMs, with per-model win rates spanning significant human advantage to numerical LLM advantage — a narrower finding that bears on this claim
• On codes requiring recognition of pedagogical intent not explicitly stated, LLMs over-apply and the blind verifier rejects their application in favor of human coding — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related
• CLARA annotations align with blinded educator judgments, with 0.81 pairwise agreement and highest-rated educational interpretability — related
• CLARA achieves the highest agreement with blinded human developmental ranking judgments compared with readability and prompting baselines — related
• LLM and human written 7C analyses show no overall difference in behavioral alignment or evidence correspondence, but align less on Communication and Constructive dimensions — related
• Human raters' preferences align with the LLM judge on TutorBench (Pearson r = 0.82 across metric-level win rates) — related
• On structured pedagogical-judgment items, all evaluated models share a systematic style-over-fit deviation, converging on the same non-reference option — related
• The cross-family LLM judge panel agrees with human domain experts on every judged axis, reaching the expert ceiling on answer-holding — related
• Humans are more certain when judging that facilitation is not needed than when judging it is — related
• Hierarchical Acceptance Rate aligns with expert acceptance decisions better than single-dimension and holistic LLM judges — related
• Blinded expert annotators misclassified 83.7% of LLM-generated errors as human-written — related
• Automated metrics and LLM judges can misalign with expert human preferences in non-verifiable educational evaluation — related
• Students engaged deeply with LLM feedback, with coded evidence of reflection, rebuttal, and acceptance in the large majority of responses — related
• The LLM judge agrees with human annotators at 87.7% (κ = 0.842) on consensus pairs, but severity-rating agreement is uneven and medium-tier boundaries are uncertain