An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Inter-expert Kappa for coding conflict resolution and project management was 0.45-0.64, and LLM-expert agreement was similar to that between the two experts. → Globerson 2026

Evidence¶

Globerson 2026¶

Globerson, A., Keeling, A., Choudhury, A., et al. (2026). Towards Scalable Measurement of Durable Skills. https://arxiv.org/abs/2609.15864 doi:10.48550/arxiv.2609.15864

q2 · i? · causal · r2

Evaluation of the Vantage collaboration protocol with 188 Prolific-recruited participants generating 373 conversations, rated by two NYU pedagogical raters and a Gemini-based AI Evaluator. Cohen's Kappa inter-expert agreement was "in the range of 0.45-0.64", and LLM-expert agreement was similar, supporting automatic rating as a scalable alternative.

"Agreement between the human expert raters was moderate, with Kappa in the range of 0.45-0.64 [28]. Agreement between the LLM and the human experts was similar to that between the two experts."

Discussion¶

Related Claims¶

• Back-translation evaluation outperforms LLM-as-a-Judge (code+image) in agreement with human raters across four models — related
• A Gemini-based creativity autorater scores real students' complex multimedia creativity tasks on par with human experts (item Kappa 0.66; total-score Pearson r = 0.88) — related
• LLM-based transcript classifications agreed highly with human judgment for explicit questions (κ = .778) and math talk (κ = .722), but only moderately for need for help (κ = .562) and confusion (κ = .531) — related
• LLM-generated 7C collaboration assessment scores fall within the range of human expert variability across ten discussions — related
• The cross-family LLM judge panel agrees with human domain experts on every judged axis, reaching the expert ceiling on answer-holding — related
• LLM ratings of classroom transcripts are more correlated with each other than with expert human ratings, across the same and different tasks — related
• Gemini-2.5-Pro can effectively replicate human expert judgments when answering SLM-generated MCQs — related
• Human inter-rater agreement on the writing rubric is imperfect: no rubric item reaches full agreement and the most discrepant rater generally falls below 80% — related
• LLM-generated misconceptions grounded in quiz content, response patterns, and lecture transcripts are rated excellent by subject matter experts — related
• Inter-model agreement on LLM skill tagging is moderate (Cohen's kappa 0.51), with strong agreement on S5 and disagreement concentrated in the S2/S3/S4 boundary — related
• Expert tutors' lesson-planning and practice-design leaderboards separated AI models, and expert-rated practice design correlated with fewer student answer disputes — related
• The star-to-sentiment labeling rule agrees with human review-level labels at Cohen's kappa 0.66 with 82% accuracy on 3,000 RateMyProfessor teacher reviews — related
• Inter-rater agreement among the 20 expert teachers was low, at or below chance across QM criteria — related
• LLM-based rubric scores agreed with expert ratings at the total-score level and on cognitive-process dimensions, but agreement was low for more subjective dimensions — related
• Out-of-the-box LLMs agree with human essay grades far less than human raters agree with each other — related
• AI assessors show emerging reliability: acceptable AI-human rater agreement (Cohen's 0.78) in a 12-educator case study and human-comparable feedback helpfulness ratings with 45 learners — a narrower finding that bears on this claim
• LLM-based scoring of Spanish argumentative writing achieves 60–80% per-item accuracy, slightly below human inter-rater agreement — related
• NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• LLM observer ratings converge excellently with human expert judgments in applying the CAARS observer instrument to generated narratives — a narrower finding that bears on this claim
• Inter-rater agreement among pre-service teachers is low regardless of grading approach or essay quality — related
• Five judges classifying 63 test units reached average inter-judge correlation of 0.542 ± 0.019 on the two scales — related
• Judges' ratings on the Thorndike handwriting scale agree closely, and averaged ratings give a constant determination — related