LLM alignment with expert teaching ratings does not predict, and is often negatively associated with, alignment with student learning gains¶

Claim · All claims
Evidence · 1 study · associational r3 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Models more aligned to expert observation ratings are not correspondingly aligned to value-added learning gains, and in many cases are negatively associated with them. → Michael Hardy 2026

Evidence¶

Michael Hardy 2026¶

Michael Hardy, Yunsung Kim. (2026). Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact. arXiv. https://arxiv.org/abs/2603.00883 doi:10.48550/arxiv.2603.00883

q2 · i? · associational · r3

Kendall's tau concordance analysis juxtaposing each LLM's alignment with expert MQI/CLASS ratings (x-axis) against alignment with stacked VAMs (y-axis) per task in Fig. 4. The article names this a "systematic disconnect" between the two alignment axes.

"Models that appear more aligned to expert judgments arenotcorrespond- ingly aligned to learning, and in many cases, are"

Discussion¶

Related Claims¶

• ML-based scoring approaches more often overestimated expert-assigned scores, whereas LLM-based approaches more often underestimated them — related
• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related
• Human raters' preferences align with the LLM judge on TutorBench (Pearson r = 0.82 across metric-level win rates) — reports the opposite
• Confidence-aware selective test-time scoring achieves the best average agreement with expert rubric scoring across six NGSS drawing items — reports the opposite
• LLM-generated difficulty ratings show only moderate rank alignment with empirical student performance on basic arithmetic items — related