The revised Round II grading workflow reduced AI over-awarding and total-score MAD, most clearly for OE1 (7.1% to 4.8%) and QM (9.4% to 3.8%)¶

Claim · All claims
Evidence · 1 study · design r3 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Round I AI scores were positively shifted (D positive in all four comparisons); Round II reduced this shift and lowered total-score MAD for OE1 and QM. → Praveen Pathak 2026

Evidence¶

Praveen Pathak 2026¶

Praveen Pathak, Siddharth Tiwary, Charudatt Kadolkar, Vijay Singh, David Rakestraw, Shirish Pathare, and Anwesh Mazumdar. (2026). Large-scale AI grading of handwritten physics assessments: Score agreement and Olympiad team selection outcomes. https://arxiv.org/abs/2608.20521 doi:10.48550/arxiv.2608.20521

q2 · i? · design · r3

Comparison of the two full grading rounds (GPT-5.5 Thinking at high effort) on the same submissions. The article reports "OE1 MAD fell from 7.1% to 4.8%, and QM MAD fell from 9.4% to 3.8%"; no effect size is printed for this change.

"The clearest reductions in total-score MAD occurred for OE1 and QM: OE1 MAD fell from 7.1% to 4.8%, and QM MAD fell from 9.4% to 3.8%."

Discussion¶

Related Claims¶

• LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — the general claim this finding bears on, with the studies behind it
• AI total scores correlate strongly with official human scores on handwritten physics assessments (r = 0.91–0.97 in Round I; 0.93–0.96 in Round II) — related
• Focused rubric refinement that made required physics and scoring conditions explicit reduced AI–official disagreements on targeted questions (e.g., Grover item MAD 1.1 to 0.4 marks, r 0.70 to 0.82) — related
• Round II raised exact question-part agreement from about 63% to about 70% and cut parts differing by more than one point from about 13% to 7%, while exact partial-credit scoring remained hardest — related
• Simulated Human-in-the-Loop workflow reduces required full human scoring of written productions by at least 50% — related
• AI grading recovered the same five-student International Physics Olympiad team as official human grading in both rounds, though with a different internal ranking — related
• Human and agent framework revisions concentrate on the mid-level engagement categories (Levels 3–5), indicating greater ambiguity there than at scale endpoints — related
• The omnibus ANOVA on total QM score did not reach conventional significance, though a Friedman test suggested an overall difference with no significant pairwise Wilcoxon comparisons — related
• AI self-reported confidence flags identify parts with smaller grading differences, but some high-confidence parts still differ from official scores by more than one point — related
• AI-assigned total scores on a handwritten chemistry exam align strongly with TA-assigned totals (R2 = 0.91), while per-problem agreement is lower (R2 = 0.61–0.85) — related