In a DESI reanalysis, the ANCOVA approach yielded a non-significant effect of teacher supportiveness on listening comprehension while the CHANGE approach yielded a larger, significant effect¶

Claim · All claims
Evidence · 1 study (2 entries) · theoretical r3 · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Under the ANCOVA model, the regression coefficient of listening comprehension at T2 on teacher supportiveness was not statistically significant (βCA = 0.034, p = .059). → Köhler 2021 q2 i? Under the CHANGE approach, the regression of the change-score on teacher supportiveness was about 30% larger and statistically significant (βCS = 0.090, p = .002). → Köhler 2021 (2)

Evidence¶

Köhler 2021¶

Köhler, C., Hartig, J., & Naumann, A. (2021). Detecting Instruction Effects—Deciding Between Covariance Analytical and Change-Score Approach. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09590-6

q2 · i? · theoretical · r3

Secondary analysis of DESI study data (N = 10,985 students in 427 classes) using doubly latent multilevel models in Mplus 7.4 with FIML for missing data. The article reports the ANCOVA coefficient was "not statistically significant (βCA = 0.034, SE = 0.018, p = .059)".

"under the ANCOVA model, the regression coefficient when regressing listening comprehension at T2 on teacher supportiveness was not statistically significant (βCA = 0.034, SE = 0.018, p = .059)"

Köhler 2021 (2)¶

Köhler, C., Hartig, J., & Naumann, A. (2021). Detecting Instruction Effects—Deciding Between Covariance Analytical and Change-Score Approach. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09590-6

q2 · i? · theoretical · r3

Same DESI multilevel latent variable analysis, CHANGE approach with a latent change-score variable ΔLC at both levels. The article reports the coefficient was "about 30% larger and statistically significant ( βCS = 0.090, SE = 0.029, p = .002)".

"In the change-score approach, however, the regression of the change-score on teacher supportiveness was about 30% larger and statistically significant ( βCS = 0.090, SE = 0.029, p = .002)"

Discussion¶

Related Claims¶

• For the DESI example, both approaches would show negative bias under the assumed data-generating model, so the observed positive coefficients make a positive true effect of teacher support likely, though its exact size remains inestimable — related
• In nonrandom studies where trait variables influence instruction, the CHANGE approach is unbiased when change is unrelated to stable skill differences, the ANCOVA overestimates, and neither is unbiased when change depends on the trait — a broader claim this one bears on
• The ANCOVA and CHANGE models can be transformed into one another, so empirical model fit is no criterion for choosing between them — related