Evaluate AI-in-LMS deployments with cluster-randomized or stepped-wedge rollout linked to multi-year institutional outcomes¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article's evaluation framework connects middle-school learning traces to medium- and long-term outcomes. Where feasible it uses cluster randomization at classroom or school level; otherwise 'the study uses a stepped-wedge design in which all participating schools eventually receive the AI-integrated LMS, but adoption is staggered across multiple waves'. Outcomes are organized in a time-horizon structure (short-term mastery and engagement, medium-term high-school trajectories, long-term postsecondary transitions), with fidelity measured from system logs and hierarchical/growth models for clustered longitudinal data.

Design Implications¶

Context¶

Requirements¶

• Implementation fidelity measurement from system logs (usage, completion, dashboard interaction)
• Stable, privacy-preserving identifiers for multi-year linkage
• Attrition handling such as inverse probability weighting and multiple imputation

Constraints¶

• Long-term postsecondary outcomes depend on data-sharing agreements; contamination, novelty effects, implementation variability, and measurement drift are anticipated threats requiring mitigation

Target Learners¶

• Middle school student cohorts followed through high school and beyond

Target Learning Goals¶

• Durable academic gains persisting across educational transitions

Related Strategies¶

• Evaluate AI systems before deploying them with students

Examples¶

-

Key Sources¶

• Misan Paul Etchie, Olutosin Taiwo. (2026). AI-Integrated Learning Management System for Middle School: A Longitudinal Study of Learning Outcomes Through High School and Beyond. https://arxiv.org/abs/2606.07544