In two secondary-school writing studies, peer reviewers' own performance predicted how helpful their feedback was but not how accurately it identified problems¶

Claim · All claims
Evidence · 2 studies · 2 associational r2 · q2
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person

In the two correlational studies recorded here, both with U.S. secondary-school writers, a peer reviewer's own writing performance predicted whether their comments were rated helpful, but not whether they correctly identified real problems; and peer ratings made with a well-specified rubric were about as reliable and valid in lower- as in higher-performing schools. The page concerns the accuracy and helpfulness of feedback given, not the benefit of receiving or producing it. Neither study manipulated expertise, so neither shows a causal effect.

Subclaims¶

q2 i? Among secondary-school peer assessors, writer/assessor performance level was unrelated to whether they correctly identified a genuine problem in a peer's draft, but it strongly predicted whether their comments were rated helpful — so "expertise" tracks the usefulness of feedback more reliably than its diagnostic accuracy. → Wu & Schunn 2023

q2 i? Among AP English students using a structured, well-specified rubric, agreement between peer ratings and teacher/expert AP scores did not differ significantly between higher- and lower-performing schools, qualifying a simple novice-vs-expert accuracy story when reviewers are given strong scaffolding. → Schunn et al. 2016

Evidence¶

Wu & Schunn 2023¶

Wu, Y., & Schunn, C. D. (2023). Assessor writing performance on peer feedback: Exploring the relation between assessor writing performance, problem identification accuracy, and helpfulness of peer feedback. Journal of Educational Psychology, 115(1), 118–142. doi:10.1037/edu0000768

q2 · observational/correlational study with regression controls (not an experimental manipulation of expertise) · i? · no d/r-scale effect size reported; findings expressed as odds ratios and null regression coefficients · n=234 assessors (1,921 individual comments), 234 secondary/high-school students in a U.S. writing course · associational · r?

234 high-school assessors' own writing performance was measured at three grain sizes (genre, rubric dimension, specific problem topic) and related to (a) whether they correctly flagged real problems in a peer's essay and (b) whether their comments were rated helpful. Multiple regression showed assessor writing performance was not related to problem-identification accuracy at any grain size — lower-performing assessors even flagged more problems (not fewer) on several topics, with directionally higher false-alarm rates that were not statistically significant. By contrast, assessor performance on specific topics and dimensions consistently and substantially predicted feedback helpfulness (topic-level performance more than doubled the odds of helpful feedback; dimensional performance raised it by ~25%), even though lower-performing assessors rarely gave outright incorrect advice.

Schunn et al. 2016¶

Schunn, C., Godley, A., & DeMartino, S. (2016). The reliability and validity of peer review of writing in high school AP English classes. Journal of Adolescent & Adult Literacy, 60(1), 13–23. doi:10.1002/jaal.525

q2 · observational/correlational classroom study, no experimental manipulation of reviewer ability · i? · reported as correlations (r ≈ .4–.7) and a non-significant moderation test, not a d-scale effect for the expertise comparison · n=1,215 students across 26 U.S. schools (12 states); 489 essays independently rated by teachers and trained AP scorers · associational · r2

1,215 AP English students anonymously rated five classmates' rhetorical-analysis essays each with a task-specific rubric; their mean ratings were compared to their teachers' and to trained AP expert scorers' ratings of the same 489 essays. Correlations between mean student ratings and both teacher and expert scores were moderate-to-high (roughly .4–.7, approaching .7 for the overall essay score) and, notably, students' averaged ratings correlated with expert scores slightly more strongly than individual teachers' ratings did. Critically for this claim, reliability and validity did not differ significantly between higher- and lower-performing schools on any rubric criterion, suggesting that — at least when reviewers use a carefully designed, well-specified rubric — peer-assessment accuracy does not simply track the reviewers' own academic performance level.

Studies bearing on this claim¶

How accurately do learners judge work, and what does the accuracy of their judgments depend on? This table gathers the studies behind this claim and the 12 narrower claims that bear on it: 14 distinct studies (5 associational, 5 causal, 2 quant-synthesis, 1 review, 1 qualitative), q2–q4, 3 of 14 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

What the evidence shows. Wu & Schunn (2023) found assessor expertise predicted how helpful feedback was but was unrelated to how accurately it identified problems, and Schunn et al. (2016) found peer ratings about as reliable in lower- as in higher-performing schools once a well-designed rubric was used. The page's former title (that peer feedback accuracy depends on expertise) is not borne out by these studies, at least where assessment is structured; the reasoning below is the theoretical case for that expectation, which the evidence here does not confirm.

Why expertise matters. Accurate peer feedback requires the same knowledge that accurate self-assessment requires: a mental model of what quality looks like and the ability to detect deviations from it. Novices lack this model, so their comments tend to be generic ("good introduction") or wrong (praising fluent but incorrect reasoning). This is closely tied to the expertise reversal effect — the same knowledge gap that makes worked examples valuable for novices makes them unreliable evaluators of others' work. Under cognitive load theory, evaluating a peer's work demands comparing it against internal standards for quality; without those schemas, novices default to surface features.

Moderators and boundary conditions. The claim is directional, not absolute. Peer feedback can still be valuable for novices when (a) the task is well-structured with explicit criteria or rubrics that substitute for missing internal standards, (b) feedback is exchanged at a surface level (clarity, organization) rather than deep conceptual correctness, or (c) peers collectively catch errors that individuals miss. Conversely, the higher the domain-specificity of the judgment required, the more accuracy depends on expertise — a constraint that worsens as tasks become more open or conceptually demanding. This mirrors the broader finding that feedback improves achievement mainly when it tells the learner where they stand relative to goals and how to close the gap (Assessment for learning improves achievement) [+M] — a function novice peers are poorly positioned to serve on conceptually demanding work.

Design implications. Where peer feedback is used with novices, designers should treat it as a supplement to, not a substitute for, instructor or expert feedback, and should scaffold the judgment itself: rubrics, annotated exemplars contrasting strong and weak work, and training in applying criteria. Receiving peer feedback may still benefit the receiver even when the giver's comments are imperfect — the act of reviewing can activate criteria in the giver as well — but designers should not assume accuracy scales with participation. Note also that simply adding evaluation checklists does not by itself produce accurate judgments online (Checklist evaluation is ineffective online) [~M]; the scaffolds must build genuine criterion knowledge, not just prompt surface checks. Where criterion knowledge is genuinely weak, pairing peer review with worked examples or annotated models of strong and weak work is a more promising route than procedural supports alone. Structured peer-review protocols that distribute judgment across a group (e.g., calibrated peer review with exemplar comparison before rating) partially mitigate the problem, but they substitute external criteria for internal ones rather than eliminating the expertise dependence.

Open questions. Two correlational studies of secondary-school writing are recorded here; no experiment manipulating reviewer expertise, and no study outside writing, is recorded yet. The expertise threshold at which peer feedback becomes reliably accurate, if any, is not established, and evidence across domains and on less structured tasks is needed before strong design prescriptions can be made.

Learner Variables¶

• Writing Skills — predictor: learners who differ on it differ in outcomes

Related Claims¶

• Peer Feedback Improves Writing — the general claim this finding bears on, with the studies behind it

• In a re-analysis of PBL tutor-background data, neither a meta-regression nor subgroup comparisons found that tutor expertise predicts student learning — the general claim this finding bears on, with the studies behind it

• Expertise reversal effect — the same knowledge asymmetry governs when scaffolds and peer evaluation help or hurt.

• Cognitive load theory — explains why novices lack the schemas needed to evaluate work accurately.
• Assessment for learning improves achievement — feedback only helps when it accurately locates work relative to criteria, which is exactly what novice peers struggle to do.
• Checklist evaluation is ineffective online — surface-level evaluation supports do not compensate for missing criterion knowledge.
• Evidence that domain-general perceptual-cognitive skills distinguish athletes is mixed; expertise appears domain-specific — related
• The quality of feedback students give peers predicts gains in their own work, but no recorded study isolates a causal benefit of the assessor role — related
• Structured criteria in peer assessment are not shown to improve its effect on academic performance over freeform review — related
• Peer assessment improves academic performance by a small average amount relative to no assessment or to teacher assessment — related
• Prior knowledge is not significantly related to test performance in this sample — related
• In two online comparisons, a rubric changed what peer reviewers commented on (one study) and helped peer raters place writing in quality tiers, most for complex criteria (the other); the first study says the rubric restrained reflective thinking — related
• Students' application of teacher support predicts accurate answers in small-group work — related
• The report measures how novice teachers' effectiveness changes with experience relative to veteran teachers' change in effectiveness — related
• Students given AI assistance in peer feedback tended to rely on the AI-generated feedback and struggled to perform the task when the assistance was removed — related
• Students apply precision-of-language as a distinct evaluation criterion, flagging hedging phrases as open to interpretation and proposing concrete substitutions — related
• Giving students feedback about their grading bias increased their subsequent grading accuracy — related
• Students' expertise on the task may be an important prerequisite for accurate self-assessment — related
• A formative assessment script increases the quantity and quality of peer feedback vocational students generate on a peer's draft — related
• Giving feedback that identifies problems in peers' work is a significant predictor of assessors' own project performance — related
• Giving suggestions to peers is a significant predictor of assessors' own project performance — related
• Turn-taking test review makes each member's expertise visible, enabling peers to locate and draw on community knowledge — related
• Anonymous asynchronous online peer review of writing: lower-skill students sometimes felt inferior yet increased determination, while better writers felt they learned less — related
• First-year students do not attach great importance to the tutor's domain specific expertise, which may explain why staff tutors' extra expertise does not yield achievement advantages — related
• Students rated peers' work from their own country 3.6% higher than work from elsewhere — related
• PeerWise student ratings of multiple-choice questions correlate well with instructor ratings (0.5 and 0.58) — related
• Peers outperformed ChatGPT in identifying problems within essays — related
• Peer ratings of programming assignments correlated significantly with instructor ratings, with students more accurate on objective than subjective judgements — related
• Most studies used online peer feedback to improve students' task performance — related