MAS-CMD was rated significantly higher than MAS-Roles on QM criterion 5.2C active learning by 20 expert teachers, with a large effect size¶

Claim · All claims
Evidence · 1 study · design r2 · q3 · i3 large
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i3 In a within-subject human evaluation, teachers rated MAS-CMD significantly higher than MAS-Roles on QM 5.2C active learning (F(2,38)=4.75, p=.014, η²p=.20). → Jiayi Wang 2025

Evidence¶

Jiayi Wang 2025¶

Jiayi Wang, Ruiwei Xiao, Xinying Hou, John Stamper. (2025). Enabling Multi-Agent Systems as Learning Designers: Applying Learning Sciences to AI Instructional Design. https://arxiv.org/abs/2508.16659 doi:10.48550/arxiv.2508.16659

q3 · i3 · design · r2

Human expert evaluation in which 20 US secondary math and science teachers rated all three systems on QM criteria using repeated-measures ANOVA. The article reports "a statistically significant effect of system on criterion 5.2" with η²p = .20, described as a large effect size.

"ANOVA revealed a statistically significant effect of system on criterion 5.2, F (2, 38) = 4.75,p = .014,η2 p = .20, indicating a large effect size."

Discussion¶

Related Claims¶

• LLM-as-a-judge ratings found only small, incremental differences between systems, with ceiling effects on several criteria — related
• Inter-rater agreement among the 20 expert teachers was low, at or below chance across QM criteria — related
• Teachers qualitatively preferred MAS-CMD activities for creativity, real-world context, and classroom-ready completeness — related
• The omnibus ANOVA on total QM score did not reach conventional significance, though a Friedman test suggested an overall difference with no significant pairwise Wilcoxon comparisons — related