AI-teacher diagnostic agreement was higher in the Diagnostic Review class and varied by issue type, with local language issues best diagnosed¶

Claim · All claims
Evidence · 2 studies · 2 causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Mean issue-level F1 agreement was 0.768 in Diagnostic Review versus 0.650 in Control, and Diagnostic Review F1 rose from 0.738 in Cycle 1 to 0.803 in Cycle 3. → Yao X-C and Fan L 2026 q2 i? Cycle 3 agreement was highest for vocabulary word choice (F1 = 0.824), coherence/cohesion (F1 = 0.816), and content/argument (F1 = 0.815); the lowest Control value was source use/evidence in Cycle 1 (F1 = 0.579). → Yao X-C and Fan L 2026 (2)

Evidence¶

Yao X-C and Fan L 2026¶

Yao X-C and Fan L (2026) From automated scoring to learning diagnosis: a mechanism study of AI-supported formative assessment in English writing. Front. Psychol. 17:1905455. https://doi.org/10.3389/fpsyg.2026.1905455

q2 · i? · causal · r2

Issue-level AI-teacher agreement analysis using teacher-confirmed labels as reference, summarized in a bubble matrix by issue type and cycle. "Mean F1 was 0.650 (SD = 0.030) in Control and 0.768 (SD = 0.037) in Diagnostic Review." No standardized effect size is printed for this contrast.

"Mean F1 was 0.650 (SD = 0.030) in Control and 0.768 (SD = 0.037) in Diagnostic Review."

Yao X-C and Fan L 2026 (2)¶

Yao X-C and Fan L (2026) From automated scoring to learning diagnosis: a mechanism study of AI-supported formative assessment in English writing. Front. Psychol. 17:1905455. https://doi.org/10.3389/fpsyg.2026.1905455

q2 · i? · causal · r2

Issue-type breakdown of AI-teacher agreement across cycles. The highest Cycle 3 F1 values were for "vocabulary word choice (F1 = 0.824)" and related categories; the lowest Control value was source use/evidence in Cycle 1.

"Cycle 3 values were highest for vocabulary word choice (F1 = 0.824), coherence/cohesion (F1 = 0.816), and content/argument (F1 = 0.815); the lowest Control value was source use/evidence in Cycle 1(F1 =0.579)."

Discussion¶

Related Claims¶

• The Diagnostic Review class scored higher on diagnostic accuracy, actionability, comprehension, revision quality, and self-assessment accuracy across all three cycles — related
• Diagnostic Review revisions contained larger shares of meaning-change, organization, and content/argument work, while Control revisions remained more dependent on surface-level editing — related
• Human-LLM agreement is moderated by code properties, multi-model consensus, and model-reported confidence — related
• LLMs align better with human coding on concise theories with discrete concepts than on more complex ones — related
• Fine-tuned PLMs measure high-inference teaching practices better when the variable requires less pedagogical expertise, matching human-rater agreement on lexical variables but degrading on inference-heavy ones — related
• The Diagnostic Review class gained more writing points across three cycles than the Control class (Hedges' g = 1.12) — related
• Teacher review workload was higher in the Diagnostic Review class in review minutes, added instruction, and review intensity — related
• Diagnostic Review reflection records contained fewer misunderstanding codes and higher reflection quality than Control follow-up records — related
• Qualitative error analysis yields five recurring error types, with over-inference most prevalent (33.5%), followed by evidence misidentification (23.4%) and hallucination (19.8%) — related