Grading time shows a sweet spot: students grading around 20 minutes (time z-score near -0.25) have significantly lower residual standard deviations than slow graders, while snap graders are unreliable and inflate grades¶

Claim · All claims
Evidence · 1 study · associational r1 · q2
Currency · newest study 2013 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Students grading with time z-score around -0.25 have significantly lower residual standard deviations (p-value < 0.001, diff = 0.3 standard deviations) than students with time z-score > -0.20; snap graders (z-score < -0.30) are unreliable and tend to slightly inflate grades. → Chris Piech 2013

Evidence¶

Chris Piech 2013¶

Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng, Daphne Koller. (2013). Tuned Models of Peer Assessment in MOOCs. arXiv. https://arxiv.org/abs/1307.2579

q2 · i? · associational · r1

Observational analysis of tens of thousands of peer grades in the HCI datasets, comparing grading residual to time spent grading (Figure 4a). The article reports the sweet spot "corresponds to around 20 minutes grading" for most assignments and is only visible with normalized grading times.

"Students who grade assessments with a time that has a z-score of around -0.25 have significantly lower residual standard deviations (withp-value< 0.001, diff = 0.3 standard deviations) than students who take a long time to grade (i.e., time spent grading has a z-score > -0.20)."

Discussion¶

Related Claims¶

• Pooling Repeated LLM Grades Does Not Improve Agreement on Short Answers — related
• Higher-scoring students tend to be more reliable graders (except the very best), and students are least biased when grading peers with similar scores — related