Report distributional quantities alongside average effects when evaluating AI learning resources¶

Strategy · All strategies
Evidence · 2 claims (2 for) · 1 study (1 causal), q3 · 0 of 1 report an effect size · 2 claims rest on one study

Description¶

The article proposes that evaluations of AI learning resources report, alongside any average effect, three distributional quantities: "the change in the share of students below the grade boundaries that matter for progression and employment, the change in dispersion, and the threshold or quantile profile of the effect". The motivation is that an average treatment effect combines the outcomes of the students the resources are meant to help with everyone else's and cannot test equity claims; in this study the average effect was statistically fragile while the threshold effect was robust.

Design Implications¶

Context¶

Requirements¶

• Assessment records that institutions already hold, from which the three quantities can each be computed

Constraints¶

• The article notes meta-analyses pooling average effects inherit the same limitation, and that neither its own nor comparable estimates should be read as the effect of AI-generated podcasts in general

Target Learners¶

• all attainment levels in a course, since the point is to detect who gains and who does not

Target Learning Goals¶

• examination and course outcomes evaluated across the attainment distribution rather than in the mean

Claims¶

• Chatbot treatment effects on final grades were statistically significant between the 35th and 75th quantiles of the course grade distribution and at the 95th [+W]
• Women assigned to the chatbot in Microeconomics earned final grades seven points higher than control women, with no treatment effects for men [+W]

Related Strategies¶

Examples¶

-

Key Sources¶

• Dang, C. T., & Nguyen, A. (2026). Verified, not generated: expert-verified AI study materials and the distribution of learning gains in a university course. https://arxiv.org/abs/2610.07097. doi:10.48550/arxiv.2610.07097