Focused rubric refinement that made required physics and scoring conditions explicit reduced AI–official disagreements on targeted questions (e.g., Grover item MAD 1.1 to 0.4 marks, r 0.70 to 0.82)¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? On the refined QM Q3b Grover-diagram question, refinement reduced MAD from 1.1 to 0.4 marks, raised Pearson's r from 0.70 to 0.82, and raised exact agreement from 15% to 45%. → Praveen Pathak 2026 q2 i? On the OE1 Q2 subset with large Round I differences (n=72), clearer wording cut MAD from 38% to 17% of the part maximum and raised r from 0.58 to 0.83. → Praveen Pathak 2026 (2)

Evidence¶

Praveen Pathak 2026¶

Praveen Pathak, Siddharth Tiwary, Charudatt Kadolkar, Vijay Singh, David Rakestraw, Shirish Pathare, and Anwesh Mazumdar. (2026). Large-scale AI grading of handwritten physics assessments: Score agreement and Olympiad team selection outcomes. https://arxiv.org/abs/2608.20521 doi:10.48550/arxiv.2608.20521

q2 · i? · causal · r1

Focused refinement run on QM question 3, part b (3 points, 40 responses), regrading with explicit rubric items. The article reports the refinement "increased Pearson'srfrom 0.70 to 0.82"; r here is AI–official score agreement, not an effect size of the refinement.

"For this question, the refinement reduced MAD from 1.1 to 0.4 marks, increased Pearson'srfrom 0.70 to 0.82, raised exact agreement from 15% to 45%, and raised agreement within 0.5 mark, including exact matches, from 35% to 80%."

Praveen Pathak 2026 (2)¶

Praveen Pathak, Siddharth Tiwary, Charudatt Kadolkar, Vijay Singh, David Rakestraw, Shirish Pathare, and Anwesh Mazumdar. (2026). Large-scale AI grading of handwritten physics assessments: Score agreement and Olympiad team selection outcomes. https://arxiv.org/abs/2608.20521

q2 · i? · causal · r1

Focused regrading of only the OE1 Q2 submissions with large Round I AI–official differences (n=72), an 8-point conceptual thermodynamics question. The article reports "Pearson'srincreased from 0.58 to 0.83"; r here is AI–official score agreement, not an effect size of the refinement.

"Within this subset, MAD fell from 38% to 17% of the maximum possible score for the part, and Pearson'srincreased from 0.58 to 0.83."

Discussion¶

Related Claims¶

• Prompt quality and detailed scoring rubrics significantly influence the accuracy and consistency of LLM-generated grades — the general claim this finding bears on, with the studies behind it

• Criterion-referenced tests require empirically-based construct validation studies — the general claim this finding bears on, with the studies behind it

• AI total scores correlate strongly with official human scores on handwritten physics assessments (r = 0.91–0.97 in Round I; 0.93–0.96 in Round II) — related
• AI self-reported confidence flags identify parts with smaller grading differences, but some high-confidence parts still differ from official scores by more than one point — related
• Round II raised exact question-part agreement from about 63% to about 70% and cut parts differing by more than one point from about 13% to 7%, while exact partial-credit scoring remained hardest — a broader claim this one bears on
• The revised Round II grading workflow reduced AI over-awarding and total-score MAD, most clearly for OE1 (7.1% to 4.8%) and QM (9.4% to 3.8%) — related
• Human and agent framework revisions concentrate on the mid-level engagement categories (Levels 3–5), indicating greater ambiguity there than at scale endpoints — related
• Most AI grading errors (87% for the best model) were attributable to transcription failures rather than rubric misapplication — related
• Revising the rubric to operationalize pedagogical constructs raises human–machine agreement for openness and depth — related