GPT-generated feedback passes simulated-student helpfulness evaluations only about 35% of the time on real student data¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2025 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? When applied to real student data, only approximately 35% of GPT-generated feedback passed simulated student helpfulness evaluations. → Reddig 2025
Evidence¶
Reddig 2025¶
Reddig, J. M., Arora, A., & MacLellan, C. J. (2025). Generating In-Context, Personalized Feedback for Intelligent Tutors with Large Language Models. Int J Artif Intell Educ. https://doi.org/10.1007/s40593-025-00505-6
q2 · i? · design · r2
The article's introduction reports this result from its series of studies examining whether LLMs can autonomously evaluate the helpfulness and informativeness of the feedback they generate; the full Study 3 text was not available in the supplied excerpt.
"when applied to real student data, GPT-generated feedback only pass simulated student helpfulness evaluations approximately 35% of the time."
Discussion¶
Related Claims¶
- Immediate post-feedback improvement is near zero on average and does not differ significantly across feedback conditions — related
- Prompting-based student simulators maintain artificially high pass rates (~80%) from the first submission step, while fine-tuned models track real students' low-start, sharp-increase trajectories — related
- Human scoring and feedback for HDR takes about 11 minutes 30 seconds per learner, while GPT-based scoring and feedback is nearly instant — related
- Differences in student pass-rate distributions between test splits explain the smaller INSIDE-over-SFT improvement on test_NP — related
- Fine-tuning with INSIDE internal dialogue achieves the lowest Wasserstein distances to real student code distributions on test_OP across all action-fidelity metrics — related
- Given sufficient prompt context, GPT-4 describes student errors in a College Algebra ITS with 87.8% accuracy — related
- Students whose tutors have access to Tutor CoPilot are 4 percentage points more likely to master lesson topics (pass exit tickets) in live K-12 math tutoring — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
gpt-feedback-passes-automated-helpfulness-35-percent |
title |
GPT-generated feedback passes simulated-student helpfulness evaluations only about 35% of the time on real student data |
description |
GPT-generated feedback passes simulated-student helpfulness evaluations only about 35% of the time on real student data |
status |
review |
evidence_strength |
weak |
newest_evidence |
2025 |
source_checked |
2026-10-20 |
recheck_by |
2028 |
generated |
by process:wiki-ingest on 2026-10-12 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — reddig-2025 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.