Feedback generated by GPT-3.5 and GPT-4 was significantly more readable than instructor-generated feedback on open-ended writing tasks¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? Both GPT-3.5 and GPT-4 generated feedback rated significantly more readable than instructor feedback, with GPT-4 rated highest (mean 3.91 vs 3.21). → Dai 2024

Evidence¶

Dai 2024¶

Wei Dai, Yi-Shan Tsai, Jionghao Lin, Ahmad Aldino, Hua Jin, Tongguang Li, Dragan Gašević, Guanliang Chen. (2024). Assessing the Proficiency of Large Language Models in Automatic Feedback Generation: An Evaluation Study. Computers and Education: Artificial Intelligence, 7, 100299. Preprint submitted to Elsevier. https://doi.org/10.1016/j.caeai.2024.100299

q2 · i? · design · r2

Human-rater readability evaluation (three raters, five-point scale) of 103 pieces each of instructor, GPT-3.5 and GPT-4 feedback. Means printed in Table 4: instructor 3.21, GPT-3.5 3.76, GPT-4 3.91; "feedback generated by both GPT-based models was significantly more readable than instructor-generated feedback".

"We can see that feedback generated by both GPT-based models was significantly more readable than instructor-generated feedback."

Discussion¶

Related Claims¶

• GPT models achieved high polarity agreement with instructors only for positive feedback on the Topic aspect, and could not reliably assess the other rubric aspects — related
• Structured developmental annotation (CLARA) achieves stronger alignment with developmental references than readability-based and direct prompting baselines — related
• LLMs and instructors agree broadly on where feedback belongs but diverge on exact sentence spans, with span overlap strongly goal-dependent and LLMs highlighting more of each essay — related