Loss-function decision rule for late pretest use in education RCTs

Research Method · All research methods
Evidence · 5 claims (4 for, 1 mixed) · 3 studies (2 theoretical, 1 associational), q2 · 0 of 3 report an effect size · 4 claims rest on one study

Description

Education RCTs often use pretest-posttest designs because including pretests improves the precision of estimated treatment effects. However, when pretests are collected after random assignment, analysts should not automatically include them: the decision "involves a variance-bias trade-off" that should be evaluated using a loss function approach before late pretest data are used in impact estimation.

Accounts

• Use pretest-posttest designs in education RCTs to improve precision of estimated treatment effects, but weigh the bias risk of late pretests: Education RCTs often use pretest-posttest designs because including pretests improves the precision of estimated treatment effects. However, when pretests are collected after random assignment, analysts should not automatically include them: the decision "involves a variance-bias trade-off" that should be evaluated using a loss function approach before late pretest data are used in impact estimation. (Peter Z. Schochet (2010))
• A loss function approach grounded in the causal inference literature for evaluating late pretest use in RCTs: The article proposes evaluating the late pretest decision using a loss function approach grounded in the causal inference literature. This framework balances the precision gains from including pretest covariates against the bias risk when pretests are measured after random assignment. The article states it addresses the issue "using a loss function approach grounded in the causal inference literature," applying it to several commonly used impact estimators. (Peter Z. Schochet (2010))
• Variance-bias tradeoff framework for deciding whether to use late pretest data in education RCTs: The paper organizes the late pretest decision as a tradeoff: including late pretest data improves the precision of estimated treatment effects but risks bias because the pretest may be treatment-contaminated. The author "addresses this issue both theoretically and empirically for several commonly used impact estimators, using a loss function approach grounded in the causal inference literature." The framework thus weighs precision gains against bias risk when choosing an impact estimator. (Peter Z. Schochet (2008))

Claims

• Including late pretest data in RCT analyses can bias post-test impact estimates because pretests are collected after random assignment [+M]
• The variance-bias trade-off for late pretests is addressed both theoretically and empirically for several commonly used impact estimators [+W]
• In RCTs of interventions to improve student test scores, estimators including late pretests will typically be preferred to estimators excluding them or using other baseline data [+W]
• The late-pretest estimator preference holds as long as test score impacts do not grow very quickly early in the school year [+W] — (also cited [~W])
• Bias, not precision loss, drives the difference in proxy pre-test value-added estimates [~W]

Related Research Methods

-

Key Sources

• Peter Z. Schochet. (2010). The Late Pretest Problem in Randomized Control Trials of Education Interventions. Journal of Educational and Behavioral Statistics, vol. 35, no. 4. https://www.mathematica.org
• Peter Z. Schochet. (2010). The Late Pretest Problem in Randomized Control Trials of Education Interventions. Journal of Educational and Behavioral Statistics, vol. 35, no. 4. https://www.mathematica.org/publications/journal-article-the-late-pretest-problem-in-randomized-control-trials-of-education-interventions
• Peter Z. Schochet. (2008). The Late Pretest Problem in Randomized Control Trials of Education Interventions. Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. https://ies.ed.gov/ncee

Page metadata

Field	Value
type	research-method
id	loss-function-decision-rule-for-late-pretest-use-in-education-rcts
title	Loss-function decision rule for late pretest use in education RCTs
description	Education RCTs often use pretest-posttest designs because including pretests improves the precision of estimated treatment effects.
status	review
moved_from	theory/loss-function-approach-late-pretests, theory/variance-bias-tradeoff-late-pretests
generated	by process:settle-candidates on 2026-10-09
trust	unverified — no one has confirmed this page's content
sources	2 — peter-z-schochet-2010, peter-z-schochet-2008

Frontmatter is the page's machine-readable record — see the Schema & Guide.