GPT-4 scoring errors clustered on multiple possible answers, unrecognized vocabulary, and missed incorrect or missing answers, while faculty were overly generous graders¶

Claim · All claims
Evidence · 1 study (2 entries) · design r3 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? GPT-4 could not score accurately when students provided multiple possibilities, missed incorrect transmission probabilities, and failed to recognize accurate vocabulary absent from the rubric. → Olvet DM 2026 q3 i? Faculty errors included awarding credit for true but non-responsive answers and missing minor errors or incorrect answers. → Olvet DM 2026 (2)

Evidence¶

Olvet DM 2026¶

Olvet DM, Kruidering M, Fulton TB, Truong B, Endo K, Lucito R, Willey JM. (2026). Can Generative Artificial Intelligence Reliably Score Open-Ended Question Assessments in Undergraduate Medical Education? Medical Science Educator 36:1539–1552. https://doi.org/10.1007/s40670-026-02638-2

q3 · i? · design · r3

Error pattern analysis from the Results, in which at least two faculty reviewed responses scored differently by faculty and GPT-4, using GPT-4's generated feedback to identify reasons for discrepancies on question 1A.

"GPT-4 was unable to score accurately when students provided multiple possibilities. Also, GPT-4 often didn’t recognize when a student provided the incorrect transmission probability for Stephen’s sister or Alex."

Olvet DM 2026 (2)¶

Olvet DM, Kruidering M, Fulton TB, Truong B, Endo K, Lucito R, Willey JM. (2026). Can Generative Artificial Intelligence Reliably Score Open-Ended Question Assessments in Undergraduate Medical Education? Medical Science Educator 36:1539–1552. https://doi.org/10.1007/s40670-026-02638-2

q3 · i? · design · r3

Findings of the faculty error review in the Results section. The article also reports faculty missed minor errors and sometimes gave full credit when students provided only one of two critical answers on question 2A.

"We found that faculty were overly generous graders and awarded credit for responses that were true but did not answer the question. Students sometimes used confusing language, which also made it difficult for faculty to score."

Discussion¶

Related Claims¶

• AI grading errors concentrate in graphical tasks and include both false positives on incorrect equations and false negatives from misread sketches and labels — related
• GPT-4 generated detailed per-response feedback that was useful for identifying scoring errors and could support personalized student feedback — related
• Students encountered usability obstacles: text-recognition errors, feedback-generation delays, and overly complex vocabulary — related
• Despite generally accurate feedback, the LLM's scoring explanations could be unpredictable and prone to logical inconsistency, failing to award points even when citing the correct rubric directive — related
• Given sufficient prompt context, GPT-4 describes student errors in a College Algebra ITS with 87.8% accuracy — related