AI grading of the exam is highly stable across five independent runs at the total-score level (ICC(A,1) = 0.967), with lower but still strong cell-level stability (ICC(A,1) = 0.836)¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? On total scores per student, the AI showed high run-to-run consistency: ICC(A,1) = 0.967 for a single run and ICC(A,5) = 0.993 if five runs were averaged, with a 95% repeatability coefficient of 5.33 points. → Jan Cvengros and Gerd Kortemeyer 2026 q2 i? At the item×student level (rubric cells), agreement was lower yet still strong: ICC(A,1) = 0.836 and ICC(A,5) = 0.962, with Sw = 0.25 points and RC = 0.68 points per cell. → Jan Cvengros and Gerd Kortemeyer 2026 (2)

Evidence¶

Jan Cvengros and Gerd Kortemeyer 2026¶

Jan Cvengros and Gerd Kortemeyer. (2026). Assisting the grading of a handwritten general chemistry exam with artificial intelligence. Discover Artificial Intelligence. https://doi.org/10.1007/s44163-026-01606-4

q2 · i? · design · r2

Inter-run reliability analysis treating five independent AI grading runs as five raters, using two-way random-effects absolute-agreement ICC on raw total scores. The article also reports Sw = 1.92 points and a 95% repeatability coefficient RC = 5.33 points, with Kendall's W = 0.959 for rank stability.

"ICC(A,1)=0 .967 for a single run and ICC(A,5) =0 .993 if one were to average five runs"

Jan Cvengros and Gerd Kortemeyer 2026 (2)¶

Jan Cvengros and Gerd Kortemeyer. (2026). Assisting the grading of a handwritten general chemistry exam with artificial intelligence. Discover Artificial Intelligence. https://doi.org/10.1007/s44163-026-01606-4

q2 · i? · design · r2

Same five-run reliability analysis computed at the rubric-cell (item×student) level. The article notes cell-level jitter is small, explaining why summed totals are very stable, and cautions that with about one third of ground-truth scores at zero the ICC would be inflated.

"At the item×student level (rubric cells), agreement was lower (as expected for finer granularity), yet still strong: ICC(A,1) =0 .836 and ICC(A,5) =0 .962, with Sw =0 .25 points and RC =0 .68 points per cell"

Discussion¶

Related Claims¶

• The benchmark procedure is stable under stochastic decoding, with a maximum accuracy standard deviation of 0.43% over 20 repeated runs — related
• AI-assigned total scores on a handwritten chemistry exam align strongly with TA-assigned totals (R2 = 0.91), while per-problem agreement is lower (R2 = 0.61–0.85) — related
• Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
• Within-judge variance of the repeated LLM-judge rubric is negligible, but cross-model-family agreement remains untested — related
• Spanish Math and Reading scores show high marginal reliability and moderate-to-strong test-retest stability, lowest for fall–spring and kindergarten — related
• Multimodal LLMs grading handwritten student work achieved κ = 0.90 on arithmetic but κ ≈ 0.47 on interpreting student illustrations — related
• AI scoring consistency approaches or exceeds 90% for most rubric items, but spelling is replaced by a deterministic LanguageTool-based approach achieving 68% accuracy with 100% consistency — related
• LLM difficulty predictions for fraction items are less stable across repeated runs than for the overall item set — related
• LLM observer ratings converge excellently with human expert judgments in applying the CAARS observer instrument to generated narratives — related
• A page-by-page, rubric-guided multimodal LLM workflow graded an authentic handwritten exam in about three hours at roughly $100 in token costs, versus about $3,500 in TA time — related
• LLM-based scoring of Spanish argumentative writing achieves 60–80% per-item accuracy, slightly below human inter-rater agreement — related