Most faculty–GPT-4 scoring discrepancies were only one-point differences, shrinking across iterations¶

Claim · All claims
Evidence · 1 study · design r3 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? Across rubric types, the large majority of disagreements between faculty and GPT-4 were one-point differences, and discrepancy sizes decreased with each iteration. → Olvet DM 2026

Evidence¶

Olvet DM 2026¶

Olvet DM, Kruidering M, Fulton TB, Truong B, Endo K, Lucito R, Willey JM. (2026). Can Generative Artificial Intelligence Reliably Score Open-Ended Question Assessments in Undergraduate Medical Education? Medical Science Educator 36:1539–1552. https://doi.org/10.1007/s40670-026-02638-2

q3 · i? · design · r3

Quantification of point-value differences between graders reported in the Results and visualized in Fig. 3 for all four questions across three iterations. After iteration 3, all question 1A discrepancies were one point and question 2A had only two 2-point discrepancies.

"To establish the degree of discrepancy between faculty and GPT-4 scoring, we quantified the difference in point value between the two graders (Fig. 3). Most were only one-point differences and there were smaller discrepancies after each iteration."

Discussion¶

Related Claims¶

• LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — the general claim this finding bears on, with the studies behind it
• GPT-4 achieved almost perfect inter-rater reliability with faculty after three iterations of rubric engineering on three of four medical school OEQ questions — related
• The most challenging holistic-rubric question reached only moderate inter-rater reliability (kw=0.54) after three iterations — related
• Gaps between Cohen's κ and QWK indicate GradeOpt often assigned codes adjacent to human codes rather than entirely discrepant ones — related
• Sentence embedding models reach only moderate agreement and need human supervision for open-ended grading — related
• In one reviewed study, GPT-4 automated essay scoring showed score inflation, with AI–human rating discrepancies exceeding 1.5 standard deviations in 38% of graduate-level bibliographies — related
• Adding a model answer to the GPT-4 prompt was counterproductive, lowering scoring reliability — related