PeerWise student ratings of multiple-choice questions correlate well with instructor ratings (0.5 and 0.58)¶

Claim · All claims
Evidence · 1 study · review r2 · q2 · i3 large
Currency · newest study 2009 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2012: tool and model accuracy changes quickly

Subclaims¶

q2 i3 A validity study of PeerWise found correlations between student ratings and instructor ratings of 0.5 and 0.58, leading authors to conclude students are reasonably effective at determining quality of peer-created MCQs. → Luxton-Reilly 2009

Evidence¶

Luxton-Reilly 2009¶

Luxton-Reilly, A. (2009). A systematic review of tools that support peer assessment. Computer Science Education, 19(4), 209-232. https://doi.org/10.1080/08993400903384844

q2 · i3 · review · r2

The review reports that Denny et al. (2009) found "correlations between ratings of students and the ratings of instructors who taught the course were good (0.5 and 0.58)" for PeerWise MCQ quality ratings. Both printed correlations (r = 0.5 and r = 0.58) exceed r = .37.

"A study of the validity of the peer assessed ratings (Denny et al. 2009) found that the correlations between ratings of students and the ratings of instructors who taught the course were good (0.5 and 0.58)."

Discussion¶

Related Claims¶

• LoU Interview ratings correlated .65 with readers' ratings of written ethnographic protocols, which the author read as support for the interview's validity tempered by second-hand information — related
• In two secondary-school writing studies, peer reviewers' own performance predicted how helpful their feedback was but not how accurately it identified problems — related
• AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — related