Multimodal LLMs grading handwritten student work achieved κ = 0.90 on arithmetic but κ ≈ 0.47 on interpreting student illustrations¶
Claim · All claims
Evidence · 1 study · theoreticalr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Reliability between multimodal LLMs and expert educators dropped from κ = 0.90 on arithmetic to κ ≈ 0.47 on interpreting student drawings, showing technical skill does not equal pedagogical tacit knowledge. → Thomas 2026
Evidence¶
Thomas 2026¶
Thomas, D.R., Borchers, C., Vanacore, K.P., Koedinger, K.R., & Kizilcec, R.F. (2026). Modernizing Ground Truth: Four Shifts Toward Improving Reliability and Validity in AI in Education. https://arxiv.org/abs/2603.29141 doi:10.48550/arxiv.2603.29141
q2 · i? · theoretical · r2
The article reports, citing prior work, that multimodal math models "excel at rote arithmetic (κ= 0.90)" but "fail on conceptual illustrations (κ≈0.47)". Table 1 similarly lists κ = 0.90 (Arithmetic) and κ ≈ 0.47 (Illustrations) for multimodal LLMs grading handwritten work.
"in multimodal math, models excel at rote arithmetic (κ= 0.90) but fail on conceptual illustrations (κ≈0.47), showing technical skill does not equal pedagogical "tacit knowledge" [17]"
Discussion¶
Related Claims¶
- Structure and layout of the exam pages strongly contribute to the success of the multimodal grading method and, for now, appear to be required — related
- AI grading of the exam is highly stable across five independent runs at the total-score level (ICC(A,1) = 0.967), with lower but still strong cell-level stability (ICC(A,1) = 0.836) — related
- Confidence-aware selective test-time scoring achieves the best average agreement with expert rubric scoring across six NGSS drawing items — related
- MathCog: a benchmark of 3,036 teacher-annotated diagnostic verdicts over 639 handwritten student responses to 110 math problems with Evident/Vague evidence labels — related
- All 18 evaluated LLMs score macro F1 below 0.5 in diagnosing students' cognitive skills from handwritten math work — a broader claim this one bears on
- Multimodal input consistently improves accuracy but not F1, model size correlates moderately with performance, and reasoning models show mixed results — related
- LLMs diagnose Applying skills better than Knowing skills, but with higher evidence over-attribution and false-attribution — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
multimodal-llm-grading-kappa-arithmetic-illustrations |
title |
Multimodal LLMs grading handwritten student work achieved κ = 0.90 on arithmetic but κ ≈ 0.47 on interpreting student illustrations |
description |
Multimodal LLMs grading handwritten student work achieved κ = 0.90 on arithmetic but κ ≈ 0.47 on interpreting student illustrations |
status |
review |
evidence_strength |
weak |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — thomas-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.