GPT-4o-mini natural language feedback was overwhelmingly helpful and never harmful in quality annotation¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? 92.5% of 442 natural language feedback instances were labeled helpful, 5.4% vague, 2.0% incorrect, and none harmful. → Hasnain Heickal and Andrew Lan 2026

Evidence¶

Hasnain Heickal and Andrew Lan 2026¶

Hasnain Heickal and Andrew Lan. (2026). A Classroom Study of LLM-Generated Feedback Intervention in Introductory Programming. https://arxiv.org/abs/2606.08807 doi:10.48550/arxiv.2606.08807

q2 · i? · causal · r2

Quality evaluation of 442 natural language feedback instances using GPT-4.1 as a judge, labeling each as helpful, vague, incorrect, or harmful. The authors note "no feedback instance explicitly revealed a solution or bypassed student reasoning".

"We find that 92.5% of instances are labeled helpful (409/442), 5.4% vague (24/442), 2.0% incorrect (9/442), and none harmful."

Discussion¶

Related Claims¶

• GPT-4o's correctness-labeling errors concentrate on final turns requiring numerical calculation, and its main KC-labeling error is assigning too few standards to a turn. — related
• GPT-4o mini showed progressive turn-level convergence with accumulating context while larger models showed increasing or stable error — related
• GPT-4o's final-turn correctness labeling on CoMTA is only slightly less accurate than expert human annotators, which the authors read as close to human-level performance on a challenging task. — related
• Under FSM metrics, GPT-3.5-Turbo outperforms GPT-4o-Mini, reversing that model's higher rank on coding-oriented leaderboards — related
• Expert former math teachers rated GPT-4o's dialogue annotations very highly for student correctness and moderate-to-high for knowledge components, with volatile inter-rater reliability. — related
• Cross-model validation shows consistent degradation under multi-strategy attack across Llama-3.3-70B, GPT-4o-mini, and Claude-3.5-Haiku, with significant differences between models — related
• Gemini Flash 2.0 was selected for production over GPT-4o-mini and Llama 3.1 for its superior faithfulness and speed-to-cost ratio — related
• In the first assessment stage, peer group and ChatGPT-4o feedback showed no significant difference in quality — related
• GPT-4 generates targeted corrective feedback for 66% of student errors, but 35% of hints are too general, incorrect, or give away the answer — related
• Students rate human-generated feedback as superior to AI-generated feedback in quality, helpfulness, and clarity, and as more likely to be acted upon — related
• Natural language AI feedback is associated with higher eventual success on initially incorrect programming problems — related