Including late pretest data in RCT analyses can bias post-test impact estimates because pretests are collected after random assignment¶

Claim · All claims
Evidence · 2 studies · 2 theoretical r2 · q2
Currency · newest study 2010 · 1 of 2 entries checked against their sources (automated) · not reviewed by a person

Subclaims¶

q2 i? When pretest data are collected after random assignment for logistical reasons, including them in the analysis could bias post-test impact estimates. → Peter Z. Schochet 2010

q2 i? Because pretest data are often collected after random assignment for logistic reasons, including them in the analysis could bias post-test impact estimates. → Peter Z. Schochet 2008

Evidence¶

Peter Z. Schochet 2010¶

Peter Z. Schochet. (2010). The Late Pretest Problem in Randomized Control Trials of Education Interventions. Journal of Educational and Behavioral Statistics, vol. 35, no. 4. https://www.mathematica.org/publications/journal-article-the-late-pretest-problem-in-randomized-control-trials-of-education-interventions

q2 · i? · theoretical · r2

Theoretical analysis of the late pretest problem in education RCTs. The article states that "pretest data often are collected after random assignment, so including them in the analysis could bias post-test impact estimates."

"For logistical reasons, however, pretest data often are collected after random assignment, so including them in the analysis could bias post-test impact estimates."

Peter Z. Schochet 2008¶

Peter Z. Schochet. (2008). The Late Pretest Problem in Randomized Control Trials of Education Interventions. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee

q2 · i? · theoretical · r2

Theoretical analysis in the paper's framing of the problem: pretests collected after random assignment may be contaminated by treatment, so using them "could bias the post-test impact estimates". No effect size is reported; this is the paper's motivating analytical claim.

"For logistic reasons, however, pretest data are often collected after random assignment, so that including them in the analysis could bias the post-test impact estimates."

Discussion¶

Related Claims¶

• The variance-bias trade-off for late pretests is addressed both theoretically and empirically for several commonly used impact estimators — related
• In RCTs of interventions to improve student test scores, estimators including late pretests will typically be preferred to estimators excluding them or using other baseline data — related
• Deciding whether to collect and use late pretest data in RCTs involves a variance-bias trade-off — related
• The late-pretest estimator preference holds as long as test score impacts do not grow very quickly early in the school year — related
• Empirical examples using real-world education RCT data demonstrate the design-based grouped-data theory — related
• The RCT is considered the gold standard of research evidence because randomization ensures only the intervention could cause treatment-control outcome differences — related
• Results for high school achievement are less clear because data limitations precluded a credible impact analysis — related
• Study designs that randomly assign students within schools involve trade-offs, illustrated by the Teach For America evaluation experience — related