Probabilistic peer grading models that estimate grader bias and reliability reduce RMSE against ground-truth grades by 33% and 31% on two HCI course offerings¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2013 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2016: tool and model accuracy changes quickly

Subclaims¶

q2 i? On the first HCI offering, probabilistic models reduced RMSE on ground-truth grade prediction from 7.95 to 5.30 (33%); on the second offering from 6.43 to 4.73 (31%). → Chris Piech 2013

Evidence¶

Chris Piech 2013¶

Chris Piech, Jonathan Huang, Zhenghao Chen, Chuong Do, Andrew Ng, Daphne Koller. (2013). Tuned Models of Peer Assessment in MOOCs. arXiv. https://arxiv.org/abs/1307.2579

q2 · i? · design · r2

Simulation-based evaluation on two Coursera HCI offerings (63,199 peer grades total), comparing probabilistic models against the median-of-four-peer-grades baseline on supergraded ground-truth submissions. The article reports RMSE reductions of "33% from 7.95 to 5.30" and "31% from 6.43 to 4.73"; for the second offering, grades within 10pp rose from 88% to 97%.

"In our experiments we are able to reduce the RMS error on our prediction of the ground truth grade by 33% from 7.95 to 5.30. Similarly, on the second offering of the course we were able to reduce error by 31% from 6.43 to 4.73."

Discussion¶

Related Claims¶

• Estimating each grader's bias alone accounts for 95% of the RMSE reduction, while modeling grader reliability contributes little with only four grades per student — related
• Including grader bias and reliability as features improves prediction of future course participation from AUC 0.93 to 0.98, capturing an engagement dimension missed by assignment grade — related