Trained human evaluators provided higher-quality formative feedback than ChatGPT on four of five components, while ChatGPT outperformed humans on criteria-based feedback¶
Claim · All claims
Evidence · 1 study · causalr2·q3
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person · recheck by 2026: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? Human feedback scored higher than AI feedback on clarity of directions, accuracy, prioritization of essential features, and supportive tone (all p < .001), but AI feedback scored higher on criteria-based feedback (3.64 vs 3.40, p = .03). → Steiss 2023
Evidence¶
Steiss 2023¶
Steiss, Jacob; Tate, Tamara; Graham, Steve; Cruz, Jazmin; Hebert, Michael; Wang, Jiali; Moon, Youngsun; Tseng, Waverly; Warschauer, Mark. (2023). Comparing the Quality of Human and ChatGPT Feedback on Students' Writing. https://eric.ed.gov/?q=Comparing+the+Quality+of+Human+and+ChatGPT+Feedback+on+Students+Writing
q3 · i? · causal · r2
One-way ANOVA comparing blinded coded ratings of human and ChatGPT feedback on the same 198 secondary history essays. Human raters won four components; "AI outperformed human evaluators by .24 points on average (p = .03)" on criteria-based.
"We find that human feedback is better than AI feedback in every category except criteria-based, where AI outperformed human evaluators by .24 points on average (p = .03)."
Discussion¶
Related Claims¶
- Human evaluators' prioritization of essential features was significantly better for low-quality essays than for average- and high-scoring essays — a narrower finding that bears on this claim
- ChatGPT feedback quality varied by essay quality: accuracy declined as essay quality rose, prioritization was best for average essays, and supportive tone was lowest for low-quality essays — related
- Differences between human and ChatGPT feedback quality were small, with means within one point and partial eta-squared ranging from .01 to .12 — related
- Peers outperformed ChatGPT in identifying problems within essays — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
chatgpt-feedback-quality-versus-human-raters |
title |
Trained human evaluators provided higher-quality formative feedback than ChatGPT on four of five components, while ChatGPT outperformed humans on criteria-based feedback |
description |
Trained human evaluators provided higher-quality formative feedback than ChatGPT on four of five components, while ChatGPT outperformed humans on criteria-based feedback |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2023 |
recheck_by |
2026 |
generated |
by process:wiki-ingest on 2026-10-15 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — steiss-2023 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.