LLM ratings of classroom transcripts are more correlated with each other than with expert human ratings, across the same and different tasks¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across leading LLMs rating primary math lessons, inter-model behavioral correlations are consistently higher than correlations with expert human raters, within and across instructional tasks. → Michael Hardy 2026

Evidence¶

Michael Hardy 2026¶

Michael Hardy, Yunsung Kim. (2026). Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact. arXiv. https://arxiv.org/abs/2603.00883 doi:10.48550/arxiv.2603.00883

q2 · i? · associational · r2

Analysis of 16 LLMs' ordinal ratings of 479 lesson transcripts from 311 classrooms using bias-corrected squared distance correlation, summarized in Fig. 3. The article finds "LLM-LLM agreement is consistently higher than LLM-human agreement."

"different models' ratings are substantially more correlated with one another than with expert human ratings, both within the same task and across different in- structional tasks"

Discussion¶

Related Claims¶

• LLM-generated 7C collaboration assessment scores fall within the range of human expert variability across ten discussions — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• LLM-based transcript classifications agreed highly with human judgment for explicit questions (κ = .778) and math talk (κ = .722), but only moderately for need for help (κ = .562) and confusion (κ = .531) — related
• Expert former math teachers rated GPT-4o's dialogue annotations very highly for student correctness and moderate-to-high for knowledge components, with volatile inter-rater reliability. — related
• GPT-4o mini achieved the lowest ACT balance MAE (6.12) in replicating human supervisor ratings across 49 full transcripts — related
• Machine–machine agreement exceeds human–machine agreement but remains below human–human agreement — possibly the same claim (merge candidate)
• Expert raters rate TeachingCoach higher than a GPT-4o baseline on clarity, tone, reflection, and acknowledgment of user input — related
• Full-scale LLM versions consistently achieve higher agreement with human grades than their mini and nano counterparts — related
• Machine learning measures of student engagement correlate moderately to highly with manual ratings, but manual ratings better predict self-reported engagement — related
• The human-refined framework improved human agreement substantially but produced only modest gains for ICL-based LLM annotation — related
• GPT-4o mini showed progressive turn-level convergence with accumulating context while larger models showed increasing or stable error — related
• Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other — a broader claim this one bears on