Within-student longitudinal (fixed-effects-style) design as an approach to causal inference in edtech efficacy evaluation¶

Theory · All theories
Evidence · 3 claims (1 for, 2 mixed) · 1 study (1 associational), q2 · 0 of 1 report an effect size · 3 claims rest on one study

Description¶

The article's methodological framework: using multi-year panel data so that "each student as their own control" serves as the identification strategy, "eliminating confounding from time-invariant student factors (e.g., general ability, motivation)". The authors position this as a substantial improvement over cross-sectional between-group designs with statistical controls, and apply it to evaluate a mastery learning platform's effect on standardized test scores.

Design Implications¶

Context¶

Requirements¶

• Multi-year panel data linking individual students' platform usage and test scores across years

Constraints¶

• Cannot rule out time-varying confounds, such as hiring a tutor or a change to school policy

Target Learners¶

• U.S. students in grades 3–8 using the platform

Target Learning Objectives¶

• Estimating causal effects of platform usage on standardized mathematics achievement

Claims¶

• Mastery Platform Longitudinal Positive Effects [+M]
• Prior cross-sectional between-group research on the platform found a positive usage–test-score relationship but could not rule out unobserved confounding [~W]
• Platform treatment effects did not vary across demographic subgroups, suggesting equal benefit regardless of racial background [~W]

Related Theories¶

Examples¶

-

Key Sources¶

• Weatherholtz, K., Grimaldi, P., Yamkovenko, B., Hill, K.M. (2024). Evaluating the Impact of a Mathematics Mastery Learning Platform on Student Achievement: A Large-Scale Longitudinal Analysis. EC-TEL 2024, Lecture Notes in Computer Science, vol 15159, Springer, Cham. https://doi.org/10.1007/978-3-031-72315-5_32