LLMs diagnose Applying skills better than Knowing skills, but with higher evidence over-attribution and false-attribution¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2–r3 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Contrary to expectations, models performed better on Applying than Knowing skills in F1 (t= 4.34, p < .001) and accuracy (t= 4.85, p < .001). → Yoonsu Kim 2026 q2 i? Applying skills also exhibited higher evidence over-attribution (t= 5.44, p < .001) and false-attribution (t= 7.66, p < .001) than Knowing skills. → Yoonsu Kim 2026 (2)

Evidence¶

Yoonsu Kim 2026¶

Yoonsu Kim, Hyoungwook Jin, Hayeon Doh, Eunhye Kim, Dongyun Jung, Seungju Kim, Kiyoon Choi, Jinho Son, and Juho Kim. (2026). Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work. arXiv. https://arxiv.org/abs/2504.00843 doi:10.48550/arxiv.2504.00843

q2 · i? · design · r2

Skill-specific analysis of the 18-model MathCog evaluation (Figure 3). The article reports the Applying-versus-Knowing contrast with t= 4.34 (F1) and t= 4.85 (accuracy), both p < .001, noting the gain "came with increased evidential errors."

"models performed better onApplyingskills in bothF1 score (t= 4.34,p < .001) and accuracy (t= 4.85,p < .001). However, this performance gain came with increased evidential errors."

Yoonsu Kim 2026 (2)¶

Yoonsu Kim, Hyoungwook Jin, Hayeon Doh, Eunhye Kim, Dongyun Jung, Seungju Kim, Kiyoon Choi, Jinho Son, and Juho Kim. (2026). Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work. arXiv. https://arxiv.org/abs/2504.00843

q2 · i? · design · r3

Same skill-specific analysis. The article reports higher over-attribution (t= 5.44) and false-attribution (t= 7.66) for Applying than Knowing skills, both p < .001, indicating overstated evidential support for higher-level skills.

"Applyingskills exhib-ited higher evidence over-attribution (t= 5.44,p < .001) and false-attribution(t= 7.66,p < .001) thanKnowingskills, indicating a tendency to overstate evidential support when diagnosing higher-level cognitive skills."

Discussion¶

Related Claims¶

• GPT-4o's final-turn correctness labeling on CoMTA is only slightly less accurate than expert human annotators, which the authors read as close to human-level performance on a challenging task. — the general claim this finding bears on, with the studies behind it
• LLMs frequently over-attribute evidential strength and assert Evident evidence even for incorrect diagnoses, and the two tendencies are strongly correlated — related
• Multimodal input consistently improves accuracy but not F1, model size correlates moderately with performance, and reasoning models show mixed results — related
• Few-shot prompting does not improve diagnostic F1 or accuracy but reduces evidence over-attribution and false-attribution — related
• All 18 evaluated LLMs score macro F1 below 0.5 in diagnosing students' cognitive skills from handwritten math work — related
• MathCog: a benchmark of 3,036 teacher-annotated diagnostic verdicts over 639 handwritten student responses to 110 math problems with Evident/Vague evidence labels — related
• Multimodal LLMs grading handwritten student work achieved κ = 0.90 on arithmetic but κ ≈ 0.47 on interpreting student illustrations — related