Directly prompted Claude 3.7 macro-level deep feedback was judged effective for 93.03% of 201 paragraph-level comments¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? After third-rater resolution, 187 of 201 macro feedback comments generated by Claude 3.7 were finalized as effective (93.03%), with strong inter-rater agreement (Gwet's AC1 = 0.89). → Adnan Labib 2026

Evidence¶

Adnan Labib 2026¶

Adnan Labib, Yixuan Huang, Jiahui Wu, John Maurice Gayed, Zheng Yuan, Qiao Wang. (2026). WrAFT: a Modularized Automated Writing Evaluation System for Argumentative Essays. arXiv preprint. https://arxiv.org/abs/2607.14524 doi:10.48550/arxiv.2607.14524

q2 · i? · design · r2

Human evaluation of Claude 3.7 macro feedback on 201 paragraphs across the 40 test essays, rated for effectiveness by two expert raters with a third resolving disagreements; inter-rater AC1 was 0.89 (95% CI 0.82-0.93, p <.001).

"After a third rater resolved the disagreement of the two raters, the number of effective comments was finalized as 187 (93.03%) and that of ineffective comments, 27 (6.97%)."

Discussion¶

Related Claims¶

• Directly prompted Claude 3.7 micro-level deep feedback was judged both necessary and effective in 94.69% of 630 comments — related
• Directly prompted GPT-4o surface-level corrective edits were judged necessary and effective in 96.14% of 2049 identified edits — related
• In one essay-feedback system, fine-tuned models produced no usable deep-level feedback output, so only directly prompted LLMs were evaluated; the authors suggest prompting may be more effective than supervised fine-tuning for this task — related
• Existing AI × SEL policy claims concentrate at macro and meso levels, with almost none at the classroom level — related