Peer graders in these MOOCs score more leniently and with greater dispersion than the instructor¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Peer graders gave higher scores than the instructor in almost every case, with grade dispersion two times the instructor's in astronomy and astrobiology and four times in history and philosophy. → Chris Impey 2024

Evidence¶

Chris Impey 2024¶

Chris Impey, Matthew Wenger, Nikhil Garuda, Shahriar Golchin, Sarah Stamer. (2024). Using Large Language Models for Automated Grading of Student Writing about Science. Preprint. https://doi.org/10.21203/rs.3.rs-3962175/v1

q2 · i? · causal · r1

Comparison of instructor and peer mean grades for the same questions (Table 4), using peer grades for the full course enrollments (19,661 astronomy; 1,705 astrobiology; 113 history and philosophy).

"Peer graders give higher scores and are more lenient than the instructor in almost every case, and the dispersion in their grades is higher than the instructor dispersion, by a factor of two for the astronomy and astrobiology courses, and by a factor of four for the history and philosophy course."

Discussion¶

Related Claims¶

• For individual students, GPT-4 grades match instructor grades more closely than peer grades do, with smaller dispersion — related
• In three astronomy MOOCs, GPT-3.5 and GPT-4 grades are closer to instructor grades with lower dispersion than peer grades — related
• LLM grading performance is markedly worse for open-ended history and philosophy questions than for content-based astronomy and astrobiology questions — related
• Human Graders of Open-Ended Work Agree With Each Other Less Than Treating One Grade as Ground Truth Implies — a broader claim this one bears on