Progressively larger treatment estimates among intervention students crossing engagement thresholds are hypothesis-generating only¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Treatment estimates grew after restricting the intervention group to students crossing thresholds of 10, 20 and 30 questions, but selection into higher engagement makes causal interpretation unsafe. → Wayne Harrison 2026

Evidence¶

Wayne Harrison 2026¶

Wayne Harrison, Rahil Khowaja, Emma Dobson, Germaine Uwimpuhwe, and Steve Higgins. (2026). Evaluating AI Tutoring at the Speed of Innovation: Practitioner-Led Micro-Randomised Trials of an AI Tutoring Platform in GCSE Science. https://arxiv.org/abs/2609.14789 doi:10.48550/arxiv.2609.14789

q2 · i? · causal · r2

Threshold-based estimates from the original evaluation, re-examined in this paper. The authors state the pattern "is consistent with the hypothesis that effective use matters" but that conditioning on post-randomisation engagement does not identify a causal complier effect.

"The original evaluation additionally reported progressively larger treatment estimates after restricting the intervention group to students crossing thresholds of 10, 20 and 30 questions."

Discussion¶

Related Claims¶

• Greater Medly engagement was positively associated with post-test attainment, treated as exploratory rather than causal — related
• Multiple comparison adjustments are needed when conducting hypothesis tests across pairwise contrasts to identify the most effective interventions — related
• The dashboard's cross-sectional view compares same-grade students across terms using the full sample of testers, but attrition is not random — related
• Among students receiving at least 24 hours of Air Tutors tutoring, the program showed no significant impact on statewide summative assessments but a positive, statistically significant impact on district benchmark assessments in both districts (effect sizes 0.13 and 0.33) — related
• Allocation to Medly improved post-test GCSE science attainment relative to business-as-usual self-directed revision (Hedges' g = 0.33) — related
• No clear evidence that Medly's effect differed by Pupil Premium disadvantage status — related
• About 750 contact hours of high school ESL support can have a tangible effect toward the postsecondary threshold — related
• Guidance, immediate feedback, and system-directed control showed unadjusted moderator signals that did not survive multiplicity adjustment, making them hypothesis-generating only — related
• Intervention studies reported larger effect sizes than cross-sectional studies, and studies including more teaching assistants reported larger effects — related
• In the prior 2025 evaluation, Air Tutors showed positive short-term benchmark gains at sufficient dosage but negative short-term statewide effects in the full sample — related