Inter-rater agreement among the 20 expert teachers was low, at or below chance across QM criteria¶

Claim · All claims
Evidence · 1 study · design r2 · q3
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q3 i? Fleiss' κ across the twenty raters was slight or poor for every criterion (5.1: 0.047; 5.2: 0.053; 5.3: −0.030; 5.4: −0.043) and −0.016 overall, below chance. → Jiayi Wang 2025

Evidence¶

Jiayi Wang 2025¶

Jiayi Wang, Ruiwei Xiao, Xinying Hou, John Stamper. (2025). Enabling Multi-Agent Systems as Learning Designers: Applying Learning Sciences to AI Instructional Design. https://arxiv.org/abs/2508.16659 doi:10.48550/arxiv.2508.16659

q3 · i? · design · r2

Reliability analysis of the human expert ratings: the article reports Fleiss' κ per criterion and an overall κ of −0.016, "indicating agreement below chance level", which the authors say limited the reliability of the observed effects.

"criterion 5.1 yieldedκ = 0.047 (slight agreement), criterion 5.2 κ = 0.053 (slight), criterion 5.3 κ =−0.030 (poor), and criterion 5.4κ =−0.043 (poor)."

Discussion¶

Related Claims¶

• Criterion-referenced tests require empirically-based construct validation studies — the general claim this finding bears on, with the studies behind it
• Making trustworthiness metrics and visualizations explicit increased inter-rater reliability among learning engineers evaluating LLM responses — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• The framework shows high scoring consistency, with 79% agreement and 81% inter-rater reliability on physics questions — related
• The Reflective Judgment Model coding rubric achieves high inter-rater reliability (alpha .93) when applied to teacher educators' interview narratives — related
• Inter-annotator agreement was moderate (combined mean Fleiss's κ of 0.62 across 23 groups), yet most students reported having reached consensus — related
• An LLM-based AI Evaluator agrees with expert human raters on collaboration transcripts at a level similar to inter-expert agreement — related
• Human inter-rater agreement on the writing rubric is imperfect: no rubric item reaches full agreement and the most discrepant rater generally falls below 80% — related
• MAS-CMD was rated significantly higher than MAS-Roles on QM criterion 5.2C active learning by 20 expert teachers, with a large effect size — related
• Risk of bias among included studies was mixed, with Newcastle–Ottawa ratings ranging from 3 to 9 and high inter-rater agreement
• Inter-rater agreement among pre-service teachers is low regardless of grading approach or essay quality — possibly the same claim (merge candidate)