Agreement metrics and expert quality judgments diverge in both directions: human consensus can encode shared conservative bias that the verifier rejects in favor of LLM coding¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? For codes with moderate-to-high human-human agreement, the blind verifier more often endorses the LLM interpretation, indicating two coders can reliably agree while both under-applying a code. → Liu 2026 q2 i? For codes with low human-LLM agreement, the verifier can still prefer the LLM application, so low agreement does not guarantee the LLM interpretation is worse. → Liu 2026 (2)

Evidence¶

Liu 2026¶

Liu, A., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., He, K., & Sun, M. (2026). Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth. Preprint, no identifier printed in text. https://arxiv.org/abs/2607.28890 doi:10.48550/arxiv.2607.28890

q2 · i? · design · r2

Per-code endorsement analysis of contested codes in the blind verification protocol (median 18 contested pairs per code, percentages indicative). The verifier endorsed the LLM application of ELA Skills Development "61% of the time when an LLM applies it but 30% when a human applies it"; Unit Planning was endorsed in 79% of LLM applications while none of its three contested human applications were endorsed.

"ELA Skills Development (H-H = 0.48) is endorsed 61% of the time when an LLM applies it but 30% when a human applies it, and Entire Lesson Planning (H-H = 0.50) shows the same pattern (56% vs. 32%)."

Liu 2026 (2)¶

Liu, A., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., He, K., & Sun, M. (2026). Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth. Preprint, no identifier printed in text. https://arxiv.org/abs/2607.28890. doi:10.48550/arxiv.2607.28890

q2 · i? · associational · r2

Per-code endorsement analysis of contested codes; the article states this pattern "appears across 12 codes". For Inquiry and Deep Questions, human-LLM Jaccard was 0.15, yet the verifier endorsed the LLM application (65%) far more often than the human omission (23%), so agreement alone cannot determine automatability.

"Inquiry and Deep Questions (H-L = 0.15) shows human endorsement of 23% versus LLM endorsement of 65%. The low agreement reflects LLMs applying the code where humans do not, yet the verifier judges the LLM application appropriate far more often than the human omission."

Discussion¶

Related Claims¶

• On codes requiring recognition of pedagogical intent not explicitly stated, LLMs over-apply and the blind verifier rejects their application in favor of human coding — a narrower finding that bears on this claim
• Blind expert verification shows no overall preference for human over LLM qualitative coding when sources are judged symmetrically — related
• In blind verification, model selection matters more than source type: Bradley-Terry ranking interleaves human coders and LLMs, with per-model win rates spanning significant human advantage to numerical LLM advantage — related
• CLARA achieves the highest agreement with blinded human developmental ranking judgments compared with readability and prompting baselines — related
• On structured pedagogical-judgment items, all evaluated models share a systematic style-over-fit deviation, converging on the same non-reference option — related
• Inter-annotator agreement was moderate (combined mean Fleiss's κ of 0.62 across 23 groups), yet most students reported having reached consensus — related
• The cross-family LLM judge panel agrees with human domain experts on every judged axis, reaching the expert ceiling on answer-holding — related
• AI scoring shows a systematic conservative bias, under-grading writing so 15.3% (2024) and 16.5% (2025) of human-passing tests would fail under AI scoring — related
• Automated metrics and LLM judges can misalign with expert human preferences in non-verifiable educational evaluation — related
• During member checks, multilingual 8th-grade students contested both LLM classifications of their talk and the coding scheme itself, especially the Off-task and Claim codes — related
• The LLM judge agrees with human annotators at 87.7% (κ = 0.842) on consensus pairs, but severity-rating agreement is uneven and medium-tier boundaries are uncertain
• Judgments of comprehension in oral reading rest on unaided subjective judgment with no guarantee of consistency across times or judges — related