No clear evidence that Medly's effect differed by Pupil Premium disadvantage status¶

Claim · All claims
Evidence · 1 study · causal r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? The treatment-by-Pupil Premium status interaction was not statistically clear (0.57 marks, 95% CI -2.25 to 3.39); stratified estimates were positive for both groups. → Wayne Harrison 2026

Evidence¶

Wayne Harrison 2026¶

Wayne Harrison, Rahil Khowaja, Emma Dobson, Germaine Uwimpuhwe, and Steve Higgins. (2026). Evaluating AI Tutoring at the Speed of Innovation: Practitioner-Led Micro-Randomised Trials of an AI Tutoring Platform in GCSE Science. https://arxiv.org/abs/2609.14789 doi:10.48550/arxiv.2609.14789

q3 · i? · causal · r2

Treatment-by-status interaction analysis in the same intention-to-treat framework, using free-school-meal eligibility as a proxy. Exploratory stratified estimates were "g = 0.28 (95% CI -0.04 to 0.59)" for Pupil Premium and g = 0.35 (95% CI 0.18 to 0.52) for non-Pupil Premium students.

"There was no clear evidence that the effect of Medly differed by Pupil Premium status. The treatment-by-status interaction estimate was 0.57 marks (95% CI -2.25 to 3.39)."

Discussion¶

Learner Variables¶

• Socioeconomic Status — moderator: an instructional effect differs by group

Related Claims¶

• Effects were larger in intervention schools with a higher percentage of economically disadvantaged students or a lower percentage of White students — the general claim this finding bears on, with the studies behind it
• Allocation to Medly improved post-test GCSE science attainment relative to business-as-usual self-directed revision (Hedges' g = 0.33) — related
• Progressively larger treatment estimates among intervention students crossing engagement thresholds are hypothesis-generating only — related
• Positive subject-specific effects of Medly in Physics, Chemistry and Biology, with no evidence the effect differed across subjects — related