Triangulate internal-validity findings with interaction analyses, propensity-score generalizability indices, and adapted extensions to establish external validity¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The report proposes a three-part methodological strategy for establishing that an experimental finding generalizes when probability sampling of schools is impractical: (1) test interactions between treatment effects and policy-relevant contextual variables within the sample; (2) use propensity scores and post-stratification to quantify sample-population match and estimate a population average treatment effect; and (3) examine adapted implementations in new settings. The authors conclude that "triangulation among several approaches to examining generalizability can be a reasonable substitute for the impractical technique of probability sampling," and urge the field to balance attention to internal and external validity.

Design Implications¶

Context¶

Requirements¶

• A well-defined inference population with school-level covariate data available for the full population, such as a state longitudinal data system

Constraints¶

• Propensity methods are limited to matching variables available for the whole population and may omit factors that moderate effects, such as technology infrastructure; extension studies without local control groups support only cautious causal claims

Target Learners¶

• Education researchers and school leaders evaluating whether research findings apply to their populations

Target Learning Goals¶

• Establishing external validity and generalizability of educational intervention effects

Related Strategies¶

• Define the target population as the first step in designing a generalizable impact study
• Build research programs combining studies optimized for internal validity with studies optimized for ecological validity

Examples¶

-

Key Sources¶

• Roschelle, J., Tipton, E., Shechtman, N., Vahey, P. (2018). Generalizability of a Technology-Based Intervention to Enhance Conceptual Understanding in Mathematics (SimCalc Technical Report 10). Menlo Park, CA: SRI International. https://digitalpromise.dspacedirect.org/items/7a123d3c-22ec-46db-b7cf-709c3aa65583