In nonrandom studies where trait variables influence instruction, the CHANGE approach is unbiased when change is unrelated to stable skill differences, the ANCOVA overestimates, and neither is unbiased when change depends on the trait¶

Claim · All claims
Evidence · 1 study (3 entries) · theoretical r3 · q2
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? If average change in achievement is unrelated to stable skill differences, the CHANGE approach gives the correct instruction effect because stable skill effects cancel out. → Köhler 2021 q2 i? Under that scenario the ANCOVA approach overestimates the instruction effect when both the trait effect on instruction and δ are positive. → Köhler 2021 (2) q2 i? If the average change in achievement is related to baseline achievement, neither approach reports the correct instruction effect. → Köhler 2021 (3)

Evidence¶

Köhler 2021¶

Köhler, C., Hartig, J., & Naumann, A. (2021). Detecting Instruction Effects—Deciding Between Covariance Analytical and Change-Score Approach. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09590-6

q2 · i? · theoretical · r3

Analytical deduction for the third data-generating model in Fig. 4, illustrated with the Kunter et al. (2013) PCK study in the German tracked school system. The article reasons the stable skill effect "is constant for the outcome at T1 and T2 and thus cancels out".

"the CHANGE approach gives the correct estimate of the instruction effect, because the effect of the stable skill differences is constant for the outcome at T1 and T2 and thus cancels out"

Köhler 2021 (2)¶

Köhler, C., Hartig, J., & Naumann, A. (2021). Detecting Instruction Effects—Deciding Between Covariance Analytical and Change-Score Approach. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09590-6

q2 · i? · theoretical · r3

Analytical deduction for the ANCOVA approach under the same Fig. 4 third-picture scenario: the mixture of stable- and time-specific effects in the outcome at T1 leads to an imprecise amount of partialled variance, "leading to an overestimation of the instruction effect" under positive trait and δ effects.

"leading to an overestimation of the instruction effect (in the case that in the true data-generating model, both the effect of the trait on instruction and δ are positive)"

Köhler 2021 (3)¶

Köhler, C., Hartig, J., & Naumann, A. (2021). Detecting Instruction Effects—Deciding Between Covariance Analytical and Change-Score Approach. Educational Psychology Review. https://doi.org/10.1007/s10648-020-09590-6

q2 · i? · theoretical · r3

Analytical deduction for the fourth picture of Fig. 4, where initial mathematical skill level additionally influences the gain from T1 to T2. The article concludes "neither of the approaches reports the correct instruction effect" in that scenario.

"If the data-generating model of picture four is the true model, neither of the approaches reports the correct instruction effect."

Discussion¶

Related Claims¶

• In a DESI reanalysis, the ANCOVA approach yielded a non-significant effect of teacher supportiveness on listening comprehension while the CHANGE approach yielded a larger, significant effect — a narrower finding that bears on this claim
• For the DESI example, both approaches would show negative bias under the assumed data-generating model, so the observed positive coefficients make a positive true effect of teacher support likely, though its exact size remains inestimable — related
• In random experiments, both the ANCOVA and the CHANGE approach estimate the instruction effect without bias — related
• In nonrandom studies where state-specific variables influence instruction, neither the ANCOVA nor the CHANGE approach reports the correct instruction effect — related
• In nonrandom studies where both trait and state variables influence instruction, neither approach reports an unbiased instruction effect — a broader claim this one bears on