GPT-4 grading agrees best with the instructor when the prompt includes the instructor's model answer with a rubric, and worst with an AI-generated rubric alone¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Worst GPT-4 performance occurred when the prompt was solely an AI-generated rubric, and the authors state a model answer is crucial for reliability. → Chris Impey 2024

Evidence¶

Chris Impey 2024¶

Chris Impey, Matthew Wenger, Nikhil Garuda, Shahriar Golchin, Sarah Stamer. (2024). Using Large Language Models for Automated Grading of Student Writing about Science. Preprint. https://doi.org/10.21203/rs.3.rs-3962175/v1

q2 · i? · causal · r1

Results analysis of prompt conditions; the authors add that relying solely on rubrics may not be sufficient and the presence of a master answer is crucial for grading reliability.

"Notably, the worst performance for GPT-4 is seen when the prompt is solely an AI-generated rubric, followed by the situation where the prompt includes only the instructor’s model answer."

Discussion¶

Related Claims¶

• GPT-4 generally outperforms GPT-3.5 in matching instructor grades, with lower internal variance — related
• Prompt quality and detailed scoring rubrics significantly influence the accuracy and consistency of LLM-generated grades — a broader claim this one bears on
• Adding a model answer to the GPT-4 prompt was counterproductive, lowering scoring reliability — reports the opposite