Preregister the assumed data-generating model and use design actions to minimize bias in estimated instruction effects¶

Strategy · All strategies
Evidence · 7 claims (7 for) · 1 study (1 theoretical), q2 · 0 of 1 report an effect size · 7 claims rest on one study

Description¶

The article recommends that researchers, before analyzing two-occasion nonrandomized data, debate for each examined variable constellation what the data-generating scenario probably looked like and which assumptions likely hold. The authors state: "We advocate a thorough preregistration of the assumed data-generating model, which will not only increase a deep reflection of the theory and help decide on an adequate modeling approach, it also has the potential of enhancing transparency and reproducibility." Where both approaches are biased, the data-generating model allows approximating the size and direction of the bias to guide interpretation.

Design Implications¶

Context¶

Requirements¶

• Assumptions about the data-generating model must be argued from the study design, logical conclusions, and empirical findings from studies involving the same or related constructs.

Constraints¶

• The article notes the true size of the effect and a test for its statistical significance remain inestimable due to too many unknown parameters when both approaches are biased.

Target Learners¶

• educational and instructional researchers

Target Learning Goals¶

• selecting an appropriate analysis model and minimizing bias in instruction-effect estimates

Affordances¶

• Data Generating Model Trait State Instruction Effect

Claims¶

• In a DESI reanalysis, the ANCOVA approach yielded a non-significant effect of teacher supportiveness on listening comprehension while the CHANGE approach yielded a larger, significant effect [+W]
• For the DESI example, both approaches would show negative bias under the assumed data-generating model, so the observed positive coefficients make a positive true effect of teacher support likely, though its exact size remains inestimable [+W]
• The ANCOVA and CHANGE models can be transformed into one another, so empirical model fit is no criterion for choosing between them [+W]
• In random experiments, both the ANCOVA and the CHANGE approach estimate the instruction effect without bias [+W]
• In nonrandom studies where state-specific variables influence instruction, neither the ANCOVA nor the CHANGE approach reports the correct instruction effect [+W]
• In nonrandom studies where both trait and state variables influence instruction, neither approach reports an unbiased instruction effect [+W]
• In nonrandom studies where trait variables influence instruction, the CHANGE approach is unbiased when change is unrelated to stable skill differences, the ANCOVA overestimates, and neither is unbiased when change depends on the trait [+W]

Related Strategies¶

Examples¶

-

Key Sources¶

• Köhler, C., Hartig, J., & Naumann, A. (2021). Detecting Instruction Effects—Deciding Between Covariance Analytical and Change-Score Approach. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09590-6