Aggregating peer grades with the median improves agreement with staff grades compared with unaggregated peer grades¶

Claim · All claims
Evidence · 1 study · design r3 · q2
Currency · newest study 2015 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Unaggregated peer grades agreed with staff grades far less often (26.3% within 5%, 46.7% within 10%) than median peer grades (42.0% and 65.5%). → Kulkarni 2015

Evidence¶

Kulkarni 2015¶

Kulkarni, C., Koh Pang Wei, Le, H., Chia, D., Papadopoulos, K., Cheng, J., Koller, D., Klemmer, S. R. (2015). Peer and Self Assessment in Massive Online Classes. In: Design Thinking Research, Understanding Innovation. Springer, Cham. https://doi.org/10.1007/978-3-319-06823-7_9

q2 · i? · design · r3

Appendix analysis comparing unaggregated peer grades with staff grades in the online class. The paper reports "26.3 % of grades were within 5 % of staff grades, and 46.7 % within 10 %", versus median agreement of 42.0% and 65.5%, demonstrating the value of aggregating peer grades.

"26.3 % of grades were within 5 % of staff grades, and 46.7 % within 10 %. (Recall that the median agreement was 42.0 % and 65.5 %, respectively)."

Discussion¶

Related Claims¶

• In two secondary-school writing studies, peer reviewers' own performance predicted how helpful their feedback was but not how accurately it identified problems — the general claim this finding bears on, with the studies behind it
• In a large online class, peer grades correlated highly with staff-assigned grades, with median agreement of 42.9% within 5% and 65.5% within 10% of staff grades in the second iteration — related