In three astronomy MOOCs, GPT-3.5 and GPT-4 grades are closer to instructor grades with lower dispersion than peer grades¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r1 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Across three MOOC courses, both LLMs produced grades closer to instructor grades and with lower dispersion than peer grades. → Chris Impey 2024 q2 i? Normalized-score scatter between instructor and GPT-4 was smaller than between instructor and peer graders in all three courses (15% vs 27%, 17% vs 22%, 26% vs 38%). → Chris Impey 2024 (2)

Evidence¶

Chris Impey 2024¶

Chris Impey, Matthew Wenger, Nikhil Garuda, Shahriar Golchin, Sarah Stamer. (2024). Using Large Language Models for Automated Grading of Student Writing about Science. Preprint. https://doi.org/10.21203/rs.3.rs-3962175/v1

q2 · i? · causal · r1

Three-way comparison of instructor, peer, and LLM grades across the astronomy, astrobiology, and history and philosophy MOOCs; the article reports both LLMs "closer to the instructor grades and have lower dispersion than peer grades" in all three courses.

"Summarizing the three-way comparison between instructor grades, peer review grades, and the grades assigned by the LLMs, both GPT-3.5 and GPT-4 generally produce grades that are closer to the instructor grades and have lower dispersion than peer grades. This is true for all three courses."

Chris Impey 2024 (2)¶

Chris Impey, Matthew Wenger, Nikhil Garuda, Shahriar Golchin, Sarah Stamer. (2024). Using Large Language Models for Automated Grading of Student Writing about Science. Preprint. https://doi.org/10.21203/rs.3.rs-3962175/v1

q2 · i? · design · r2

Normalized-score comparison reported in the results section: instructor-to-GPT-4 scatter of 15% vs instructor-to-peer 27% (astronomy) and 17% vs 22% (astrobiology); the history and philosophy course showed 26% vs 38%.

"For the introductory course, the mean scatter in normalized scores is 15% between the instructor and GPT-4, as opposed to 27% between the instructor and peer graders. For the astrobiology course, the mean scatter in normalized scores is 17% between instructor and GPT-4, as opposed to 22% between instructor and peer graders."

Discussion¶

Related Claims¶

• For individual students, GPT-4 grades match instructor grades more closely than peer grades do, with smaller dispersion — a narrower finding that bears on this claim
• Peer graders in these MOOCs score more leniently and with greater dispersion than the instructor — related
• GPT-4 generally outperforms GPT-3.5 in matching instructor grades, with lower internal variance — a narrower finding that bears on this claim
• LLM grading performance is markedly worse for open-ended history and philosophy questions than for content-based astronomy and astrobiology questions — related