LLM diagnostic performance drops sharply when student responses contain vague rather than evident evidence¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across all models, F1 and accuracy were substantially higher on responses with evident evidence than vague evidence, with a mean F1 gap of .51 and accuracy gap of .41. → Yoonsu Kim 2026

Evidence¶

Yoonsu Kim 2026¶

Yoonsu Kim, Hyoungwook Jin, Hayeon Doh, Eunhye Kim, Dongyun Jung, Seungju Kim, Kiyoon Choi, Jinho Son, and Juho Kim. (2026). Benchmarking Large Language Models for Diagnosing Students' Cognitive Skills from Handwritten Math Work. arXiv. https://arxiv.org/abs/2504.00843 doi:10.48550/arxiv.2504.00843

q2 · i? · associational · r2

RQ2 analysis over MathCog diagnosis instances comparing performance on evident- versus vague-evidence responses across all 18 models. The article reports t= 64.60 for F1 and t= 17.80 for accuracy, both p < .001, and a "mean∆F1 =.51; mean∆Acc=.41" gap, indicating heavy reliance on explicit evidence.

"both F1 score (t= 64.60,p < .001) and accuracy (t= 17.80,p < .001) were substantially higher when student responses containedevidentevidence for a cognitive skill than when evidence wasvague"

Discussion¶

Related Claims¶

• All 18 evaluated LLMs score macro F1 below 0.5 in diagnosing students' cognitive skills from handwritten math work — related
• MathCog: a benchmark of 3,036 teacher-annotated diagnostic verdicts over 639 handwritten student responses to 110 math problems with Evident/Vague evidence labels — related
• LLMs are more stable than supervised models across dataset shifts, though they do not surpass within-dataset fine-tuned performance — related
• Few-shot prompting does not improve diagnostic F1 or accuracy but reduces evidence over-attribution and false-attribution — related