All 18 evaluated LLMs score macro F1 below 0.5 in diagnosing students' cognitive skills from handwritten math work¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Every evaluated LLM, regardless of capability, achieved macro F1 below 0.5 on MathCog cognitive skill diagnosis, despite moderate overall accuracy. → Yoonsu Kim 2026

Evidence¶

Yoonsu Kim 2026¶

Yoonsu Kim, Hyoungwook Jin, Hayeon Doh, Eunhye Kim, Dongyun Jung, Seungju Kim, Kiyoon Choi, Jinho Son, and Juho Kim. (2026). Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work. arXiv. https://arxiv.org/abs/2504.00843 doi:10.48550/arxiv.2504.00843

q2 · i? · design · r2

Benchmark evaluation of 18 closed- and open-source LLMs on MathCog against teacher ground-truth verdicts, reporting macro F1 and accuracy. The article reports "macro F1 scores below 0.5" for all models and moderate accuracy (M=.680, SD=.050) that masks diagnostic failures.

"with macro F1 scores below 0.5 (Figure 2, Table 2). While overall accuracy was relatively moderate (M=.680,SD=.050), it obscures important diagnostic failures."

Discussion¶

Related Claims¶

• LLMs diagnose Applying skills better than Knowing skills, but with higher evidence over-attribution and false-attribution — related
• LLM diagnostic performance drops sharply when student responses contain vague rather than evident evidence — related
• LLMs frequently over-attribute evidential strength and assert Evident evidence even for incorrect diagnoses, and the two tendencies are strongly correlated — related
• Multimodal input consistently improves accuracy but not F1, model size correlates moderately with performance, and reasoning models show mixed results — related
• MathCog: a benchmark of 3,036 teacher-annotated diagnostic verdicts over 639 handwritten student responses to 110 math problems with Evident/Vague evidence labels — related
• Qualitative error analysis yields five recurring error types, with over-inference most prevalent (33.5%), followed by evidence misidentification (23.4%) and hallucination (19.8%) — related
• Multimodal LLMs grading handwritten student work achieved κ = 0.90 on arithmetic but κ ≈ 0.47 on interpreting student illustrations — a narrower finding that bears on this claim
• Few-shot prompting does not improve diagnostic F1 or accuracy but reduces evidence over-attribution and false-attribution — related