AI assessors show emerging reliability: acceptable AI-human rater agreement (Cohen's 0.78) in a 12-educator case study and human-comparable feedback helpfulness ratings with 45 learners¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r? · q2
Currency · newest study 2025 · checked against its sources 2026-10-20 (automated, 2 of 2 entries) · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i? A case study with 12 educators found acceptable rater agreement (Cohen's 0.78) between AI and human scoring. → Kostopoulos 2025 q2 i? A controlled classroom deployment with 45 learners showed AI feedback helpfulness ratings comparable to human graders. → Kostopoulos 2025 (2)

Evidence¶

Kostopoulos 2025¶

Kostopoulos, G., Gkamas, V., Rigou, M., & Kotsiantis, S. (2025). Agentic AI in Education: State of the Art and Future Directions. IEEE Access. https://doi.org/10.1109/ACCESS.2025.3620473

q2 · i? · review · r2

The review reports, citing Darvishi et al., a case study with 12 educators in which agentic assessors using analytic rubrics and generative feedback achieved "acceptable rater agreement (Cohen's 0.78) between AI and human scoring". Cohen's 0.78 is an agreement statistic, not an effect size.

"Darvishi et al. [33] (case study, N = 12 educators) found acceptable rater agreement (Cohen's 0.78) between AI and human scoring."

Kostopoulos 2025 (2)¶

Kostopoulos, G., Gkamas, V., Rigou, M., & Kotsiantis, S. (2025). Agentic AI in Education: State of the Art and Future Directions. IEEE Access. https://doi.org/10.1109/ACCESS.2025.3620473

q2 · i? · causal · r?

The review reports, citing Kaldaras et al., a controlled classroom deployment with 45 learners in which AI-generated feedback received "comparable feedback helpfulness ratings to human graders". The review characterizes such systems as emerging reliability at mid-TRL ≈ 4.

"Kaldaras et al. [34] (controlled classroom deployment, N= 45 learners) showed comparable feedback helpfulness ratings to human graders."

Discussion¶

Related Claims¶

• The emergent LLM coding pipeline reaches substantial agreement with expert human coding (Cohen's κ = 0.78, raw agreement 84%) on a stratified 10% sample — related
• An automated LLM assessor rates 3E clinical communication skills with agreement comparable to individual human raters — related
• Students rate human-generated feedback as superior to AI-generated feedback in quality, helpfulness, and clarity, and as more likely to be acted upon — related
• A human assessor agreed with the majority of GPT-generated feedback comments on assessment aspects that instructors had not commented on — a narrower finding that bears on this claim
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — a broader claim this one bears on
• GPT-4 demonstrated higher reliability of feedback than GPT-3.5, with higher F1 agreement with instructors on most assessment aspects — related
• State-of-the-art conversational agents (Blender and GPT-3) replied adequately to students but were well behind human performance in helping the student (reported from a dialogue study) — related