On codes requiring recognition of pedagogical intent not explicitly stated, LLMs over-apply and the blind verifier rejects their application in favor of human coding¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? For codes such as Tiered Scaffolding, Generate Summative Assessments, and Multimedia Use for Instruction, human applications are endorsed two-to-five times more often than LLM applications. → Liu 2026 q2 i? Fifteen codes show human endorsement two to five times higher than LLM endorsement and require recognizing pedagogical intent from context, a capability current models do not reliably demonstrate. → Liu 2026 (2)

Evidence¶

Liu 2026¶

Liu, A., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., He, K., & Sun, M. (2026). Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth. Preprint, no identifier printed in text. https://arxiv.org/abs/2607.28890 doi:10.48550/arxiv.2607.28890

q2 · i? · design · r2

Per-code endorsement analysis of contested code applications in the blind verification protocol. These are codes, in the article’s words, where "LLMs over-apply and the verifier rejects their application", and they "require recognizing pedagogical intent not explicitly stated in the text".

"Tiered Scaffolding (human endorsed 86%, LLM 27%), Generate Summative Assessments (75% vs. 21%), and Multimedia Use for Instruction (71% vs. 35%) represent codes where LLMs over-apply and the verifier rejects their application."

Liu 2026 (2)¶

Liu, A., Esbenshade, L., Xiao, M., Tian, V., Zhang, Z., He, K., & Sun, M. (2026). Agreement Is Not Quality: Blind Expert Verification of Human and LLM Qualitative Coding When Human Consensus Is Not Ground Truth. Preprint, no identifier printed in text. https://arxiv.org/abs/2607.28890. doi:10.48550/arxiv.2607.28890

q2 · i? · associational · r2

Division-of-labor classification of the 72-code instrument: the Human-required category contains 15 codes. The article states these codes "require recognizing pedagogical intent from context", and that until models develop richer pedagogical reasoning they cannot be automated without quality loss.

"These codes show human endorsement two to five times higher than LLM endorsement, for example Tiered Scaffolding (86% vs. 27%) and Generate Summative Assessments (75% vs. 21%). They require recognizing pedagogical intent from context, a capability current models do not reliably demonstrate."

Discussion¶

Related Claims¶

• Agreement metrics and expert quality judgments diverge in both directions: human consensus can encode shared conservative bias that the verifier rejects in favor of LLM coding — a broader claim this one bears on
• Blind expert verification shows no overall preference for human over LLM qualitative coding when sources are judged symmetrically — related
• In blind verification, model selection matters more than source type: Bradley-Terry ranking interleaves human coders and LLMs, with per-model win rates spanning significant human advantage to numerical LLM advantage — related