Directly prompted GPT-4o surface-level corrective edits were judged necessary and effective in 96.14% of 2049 identified edits¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Expert raters judged 1970 of 2049 ERRANT-tagged edits (96.14%) both necessary and effective in the surface-level feedback test of 40 essays. → Adnan Labib 2026

Evidence¶

Adnan Labib 2026¶

Adnan Labib, Yixuan Huang, Jiahui Wu, John Maurice Gayed, Zheng Yuan, Qiao Wang. (2026). WrAFT: a Modularized Automated Writing Evaluation System for Argumentative Essays. arXiv preprint. https://arxiv.org/abs/2607.14524 doi:10.48550/arxiv.2607.14524

q2 · i? · design · r2

Human evaluation of GPT-4o corrective edits on 40 test essays, with edits aligned via ERRANT v3.0.0 and rated by two expert raters on necessity and effectiveness. The "96.14%" precision indicates the system is nearly always correct when it identifies errors.

"Human evaluation deemed 1985 edits deemed necessary (96.88%), out of which 1970 were deemed both necessary and effective (96.14%)."

Discussion¶

Related Claims¶

• Directly prompted Claude 3.7 micro-level deep feedback was judged both necessary and effective in 94.69% of 630 comments — related
• Directly prompted Claude 3.7 macro-level deep feedback was judged effective for 93.03% of 201 paragraph-level comments — related
• Making trustworthiness metrics and visualizations explicit increased inter-rater reliability among learning engineers evaluating LLM responses — a broader claim this one bears on
• In one essay-feedback system, fine-tuned models produced no usable deep-level feedback output, so only directly prompted LLMs were evaluated; the authors suggest prompting may be more effective than supervised fine-tuning for this task — related