The TA ground truth was produced under a one-rater-per-item design with co-grading, an exceptions log, instructor audit, and student review, with an estimated grading-error rate of about 0.1% in the student-inspected subsample¶
Claim · All claims
Evidence · 1 study (2 entries) · designr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Each problem or problem part was graded by a single responsible TA, with instructor co-grading of roughly the first twenty solutions per item and an exceptions log for adjudicated alternative solution paths. → Jan Cvengros and Gerd Kortemeyer 2026
q2 i? From the post-exam review period, the authors estimate a grading-error rate of approximately 0.1% in the subsample inspected by students. → Jan Cvengros and Gerd Kortemeyer 2026 (2)
Evidence¶
Jan Cvengros and Gerd Kortemeyer 2026¶
Jan Cvengros and Gerd Kortemeyer. (2026). Assisting the grading of a handwritten general chemistry exam with artificial intelligence. Discover Artificial Intelligence. https://doi.org/10.1007/s44163-026-01606-4
q2 · i? · design · r2
Ground-truth construction (Sect. 2.4): all solutions were graded by teaching assistants under instructor supervision in a "one-rater-per-item" design; an exceptions log documented adjudicated alternative solution paths, and the instructor conducted a brief audit of all graded exams.
"each problem or problem part was graded by a single responsible TA"
Jan Cvengros and Gerd Kortemeyer 2026 (2)¶
Jan Cvengros and Gerd Kortemeyer. (2026). Assisting the grading of a handwritten general chemistry exam with artificial intelligence. Discover Artificial Intelligence. https://doi.org/10.1007/s44163-026-01606-4
q2 · i? · design · r2
During the official post-exam review period only about one-third of students inspected their grading, and less than a handful of incorrect grading decisions were discovered. The article acknowledges the ground truth is best called an operational reference standard, since mistakes may still have slipped through.
"This would correspond to approximately 150 · 46 problem parts being inspected by students, which puts the grading-error rate from this sub-sample at 5/(150 · 46) ≈ 0.1%"
Discussion¶
Related Claims¶
- Three independent expert instructors reached exceptionally high inter-rater reliability when grading 1200 bash exam responses, establishing a reliable human reference standard — related
- Instructor-confirmed grading errors occurred at roughly 0.02–0.04% of rubric line items across the term — related
- Human Graders of Open-Ended Work Agree With Each Other Less Than Treating One Grade as Ground Truth Implies — related
- LLM Short-Answer Graders Agree With One Human Rater More, and the Other Less, Than the Two Raters Agree With Each Other — related
- Students reported first-pass false-positive deductions at a mean of 2.94 per term, normalising to about 0.6% of rubric items — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
ta-ground-truth-one-rater-per-item-error-rate |
title |
The TA ground truth was produced under a one-rater-per-item design with co-grading, an exceptions log, instructor audit, and student review, with an estimated grading-error rate of about 0.1% in the student-inspected subsample |
description |
The TA ground truth was produced under a one-rater-per-item design with co-grading, an exceptions log, instructor audit, and student review, with an estimated grading-error rate of about 0.1% in the student-inspected... |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
2 — jan-cvengros-and-gerd-kortemeyer-2026, jan-cvengros-and-gerd-kortemeyer-2026-2 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.