Adding a model answer to the GPT-4 prompt was counterproductive, lowering scoring reliability¶

Claim · All claims
Evidence · 1 study · design r3 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Including a model answer in the iteration 3 prompt for question 1H reduced GPT-4's scoring reliability rather than improving it. → Olvet DM 2026

Evidence¶

Olvet DM 2026¶

Olvet DM, Kruidering M, Fulton TB, Truong B, Endo K, Lucito R, Willey JM. (2026). Can Generative Artificial Intelligence Reliably Score Open-Ended Question Assessments in Undergraduate Medical Education? Medical Science Educator 36:1539–1552. https://doi.org/10.1007/s40670-026-02638-2

q2 · i? · design · r3

Authors' observation in the Discussion on the effect of adding a model answer to the prompt for question 1H. The article cites Morjaria and colleagues reporting the lowest correlation when using only a model answer or rubric plus model answer.

"Adding a model answer in iteration 3 proved to be counter- productive, resulting in a lower reliability."

Discussion¶

Related Claims¶

• LLM grading performance relative to human graders is mixed: comparable in 12 reviewed cases, worse in 11, with the strongest alignment on short, straightforward tasks — the general claim this finding bears on, with the studies behind it
• The most challenging holistic-rubric question reached only moderate inter-rater reliability (kw=0.54) after three iterations — related
• GPT-4 achieved almost perfect inter-rater reliability with faculty after three iterations of rubric engineering on three of four medical school OEQ questions — related
• Most faculty–GPT-4 scoring discrepancies were only one-point differences, shrinking across iterations — related