The most challenging holistic-rubric question reached only moderate inter-rater reliability (kw=0.54) after three iterations¶
Claim · All claims
Evidence · 1 study · designr3·q3
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? IRR for question 1H, scored with a holistic rubric and chosen as the most challenging question, improved to substantial in iteration 2 but decreased to moderate (kw=0.54) in iteration 3. → Olvet DM 2026
Evidence¶
Olvet DM 2026¶
Olvet DM, Kruidering M, Fulton TB, Truong B, Endo K, Lucito R, Willey JM. (2026). Can Generative Artificial Intelligence Reliably Score Open-Ended Question Assessments in Undergraduate Medical Education? Medical Science Educator 36:1539–1552. https://doi.org/10.1007/s40670-026-02638-2
q3 · i? · design · r3
Holistic-rubric results from the same three-iteration faculty-versus-GPT-4 comparison. The article reports question 1H never reached almost perfect agreement, and reliability declined between iterations 2 and 3.
"After refining scoring rubrics and prompts, the IRR increased for question 1H to substantial in iteration 2 (kw=0.64, 95% CI: 0.54–0.73, 63% agreement) but decreased to moderate in iteration 3 (kw=0.54, 95% CI: 0.43–0.66, 61% agreement)."
Discussion¶
Related Claims¶
- GPT-4 achieved almost perfect inter-rater reliability with faculty after three iterations of rubric engineering on three of four medical school OEQ questions — related
- Adding a model answer to the GPT-4 prompt was counterproductive, lowering scoring reliability — related
- Most faculty–GPT-4 scoring discrepancies were only one-point differences, shrinking across iterations — related
- Sentence embedding models reach only moderate agreement and need human supervision for open-ended grading — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
holistic-hardest-question-moderate-reliability |
title |
The most challenging holistic-rubric question reached only moderate inter-rater reliability (kw=0.54) after three iterations |
description |
The most challenging holistic-rubric question reached only moderate inter-rater reliability (kw=0.54) after three iterations |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
source_checked |
2026-10-20 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — olvet-dm-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.