Raters scored significantly higher than AI on Emotional Support and significantly lower than AI on Instructional Support, with no difference for Classroom Organization¶

Claim · All claims
Evidence · 1 study · design r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? Independent t-tests showed raters scored significantly higher than AI on Emotional Support (mean difference .806, Cohen's d = 1.343, p < .001) and significantly lower on Instructional Support (mean difference -.284, d = .343, p = .043); Classroom Organization showed no significant difference (d = .019). → Fong 2026

Evidence¶

Fong 2026¶

Fong, Y., Xiang, J., Chan, T.-Y. D., Lee, K., & Lau, E. Y. H. (2026). I Code or AI Code: A Comparative Evaluation of AI-rated Scores in Classroom Observations. The Education University of Hong Kong. https://arxiv.org/abs/2609.18274 doi:10.48550/arxiv.2609.18274

q3 · i? · design · r2

Independent t-tests comparing consolidated raters' and AI scores across 71 observations (df = 140, Table 4). Emotional Support: t = 8.002, p < .001, "Cohen's d = 1.343"; Instructional Support: t = -2.041, p = .043, d = .343; Classroom Organization: t = -.111, p = .912, d = .019.

"At the domain level, 25 significant differences were observed: raters tended to score higher in Emotional Support and 26 lower in Instructional Support than AI. No significant difference was found for the Classroom 27 Organization domain."

Discussion¶

Related Claims¶

• Fine-tuned PLMs measure high-inference teaching practices better when the variable requires less pedagogical expertise, matching human-rater agreement on lexical variables but degrading on inference-heavy ones — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related