Peers outperformed ChatGPT in identifying problems within essays¶

Claim · All claims
Evidence · 1 study · associational r2 · q2 · i1 small
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i1 Peer feedback scored higher than ChatGPT feedback on the identification component of feedback quality. → Banihashem 2024

Evidence¶

Banihashem 2024¶

Banihashem, S. K., Taghizadeh Kerman, N., Noroozi, O., Moon, J., & Drachsler, H. (2024). Feedback sources in essay writing: peer-generated or AI-generated feedback? International Journal of Educational Technology in Higher Education. https://doi.org/10.1186/s41239-024-00455-4

q2 · i1 · associational · r2

MANCOVA result for the identification component (Table 1): peer mean 1.52 (SD 0.49) versus ChatGPT mean 1.29 (SD 0.70), F (1, 146) = 4.38, p < 0.01, η2 = 0.02, a small effect. Students "performed better in pinpointing and identifying the issues in the feedback provided".

"Identification Peer feedback 1.52 0.49 F (1, 146) = 4.38, p < 0.01**, η2 = 0.02"

Discussion¶

Related Claims¶

• In two secondary-school writing studies, peer reviewers' own performance predicted how helpful their feedback was but not how accurately it identified problems — the general claim this finding bears on, with the studies behind it

- Peer assessment improves academic performance by a small average amount relative to no assessment or to teacher assessment — the general claim this finding bears on, with the studies behind it¶

• ChatGPT produced more descriptive feedback than peers on argumentative essays — related
• Trained human evaluators provided higher-quality formative feedback than ChatGPT on four of five components, while ChatGPT outperformed humans on criteria-based feedback — related