AI total scores correlate strongly with official human scores on handwritten physics assessments (r = 0.91–0.97 in Round I; 0.93–0.96 in Round II)¶
Claim · All claims
Evidence · 1 study · designr3·q2·i3large
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i3 Across three assessments (OE1, OE2, QM), AI total scores tracked official human scores with Pearson correlations of 0.91–0.97 in Round I and 0.93–0.96 in Round II. → Praveen Pathak 2026
Evidence¶
Praveen Pathak 2026¶
Praveen Pathak, Siddharth Tiwary, Charudatt Kadolkar, Vijay Singh, David Rakestraw, Shirish Pathare, and Anwesh Mazumdar. (2026). Large-scale AI grading of handwritten physics assessments: Score agreement and Olympiad team selection outcomes. https://arxiv.org/abs/2608.20521 doi:10.48550/arxiv.2608.20521
q2 · i3 · design · r3
Retrospective AI grading of 10,364 scanned pages from 520 handwritten submissions across OE1, OE2, and QM, each graded twice with official rubrics and no access to human marks. The article reports "The Pearson correlations are high in RI (r= 0.91–0.97) and remain high in RII (r= 0.93–0.96)."
"The Pearson correlations are high in RI (r= 0.91–0.97) and remain high in RII (r= 0.93–0.96)."
Discussion¶
Related Claims¶
- Round II raised exact question-part agreement from about 63% to about 70% and cut parts differing by more than one point from about 13% to 7%, while exact partial-credit scoring remained hardest — related
- The revised Round II grading workflow reduced AI over-awarding and total-score MAD, most clearly for OE1 (7.1% to 4.8%) and QM (9.4% to 3.8%) — related
- AI self-reported confidence flags identify parts with smaller grading differences, but some high-confidence parts still differ from official scores by more than one point — related
- Focused rubric refinement that made required physics and scoring conditions explicit reduced AI–official disagreements on targeted questions (e.g., Grover item MAD 1.1 to 0.4 marks, r 0.70 to 0.82) — related
- AI grading recovered the same five-student International Physics Olympiad team as official human grading in both rounds, though with a different internal ranking — related
- AI-assigned total scores on a handwritten chemistry exam align strongly with TA-assigned totals (R2 = 0.91), while per-problem agreement is lower (R2 = 0.61–0.85) — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
ai-human-total-score-correlation-handwritten-physics |
title |
AI total scores correlate strongly with official human scores on handwritten physics assessments (r = 0.91–0.97 in Round I; 0.93–0.96 in Round II) |
description |
AI total scores correlate strongly with official human scores on handwritten physics assessments (r = 0.91–0.97 in Round I; 0.93–0.96 in Round II) |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — praveen-pathak-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.