Conduct a smaller sequence of rapid-cycle studies that build on one another instead of planning a single large confirmatory study¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The report recommends an iterative approach in which researchers use existing data, then run smaller studies that build upon one another. An initial study may use built-in outcome measures such as time on task or performance on the next problem set to see if results trend as expected, then add more specific measures. Researchers "have acknowledged that once an intervention (or content modification) is built into the DLP, making smaller tweaks toward improvement, and then deploying those improvements to a new sample, is a relatively efficient process."

Design Implications¶

Context¶

Requirements¶

• Built-in outcome measures or existing platform data for initial cycles; willingness to iterate design elements such as outcome measures and demographic data collection

Constraints¶

• If integrating demographic data at scale is outside the DLP's current scope, a smaller pilot sample is needed first; smaller sequences allow testing technical issues before larger samples

Target Learners¶

• Education researchers running experiments in DLPs

Target Learning Goals¶

• Refining interventions and confirming planned comparisons before large-scale deployment

Related Strategies¶

• Replace monolithic comprehensive studies with smaller, agile research cycles that test one conjecture at a time
• Leverage digital learning platforms to run unobtrusive randomized experiments on language modifications in math problems
• Plan an extended feasibility phase in which research and platform teams jointly refine design, implementation, and logistics
• Phase the intervention through cohorts that move from development to usability/feasibility testing to efficacy testing
• Phased iterative testing: optimization testing before efficacy testing before statewide scale-up
• Retrain and evaluate the wellness system on locally collected, clinically validated data, then run a formal user evaluation before wider deployment
• Conduct a larger-scale, statewide study of mentoring features and retention to detect small or moderate relationships
• Study natural methods of learning and run experiments to determine how learners pass from beginner's crude efforts to expert performance
• Direct research effort toward measuring temperament and character factors, which operate largely independently of intelligence and may be more educable

Examples¶

-

Key Sources¶

• Schellinger, A., Zacamy, J., Roschelle, J., Closser, A., & Zepeda, C. D. (2024, April). Considerations for Conducting Research in Digital Learning Platforms. Digital Promise. https://doi.org/10.51388/20.500.12265/210