For the DESI example, both approaches would show negative bias under the assumed data-generating model, so the observed positive coefficients make a positive true effect of teacher support likely, though its exact size remains inestimable¶

Claim · All claims
Evidence · 1 study · theoretical r3 · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Given the assumed data-generating model, both approaches would report a negative instruction effect if the true effect δ was zero; the observed positive coefficients therefore make a positive true effect likely. → Köhler 2021

Evidence¶

Köhler 2021¶

Köhler, C., Hartig, J., & Naumann, A. (2021). Detecting Instruction Effects—Deciding Between Covariance Analytical and Change-Score Approach. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09590-6

q2 · i? · theoretical · r3

Bias approximation using the syntax of Köhler et al. (2020a) with assumed parameter values (state variance 0.2, trait variance 0.8, trait effect on instruction −.2, trait effect on gain .3). The article concludes "it is unlikely that the true effect was zero (or negative)".

"Given the data-generating model holds, both approaches would show a negative bias and report a negative instruction effect if the true effect δ was zero. Since we found positive regression coefficients in the data, it is unlikely that the true effect was zero (or negative)."

Discussion¶

Related Claims¶

• In a DESI reanalysis, the ANCOVA approach yielded a non-significant effect of teacher supportiveness on listening comprehension while the CHANGE approach yielded a larger, significant effect — related
• In nonrandom studies where trait variables influence instruction, the CHANGE approach is unbiased when change is unrelated to stable skill differences, the ANCOVA overestimates, and neither is unbiased when change depends on the trait — related