Skip to content

Within-study comparison (WSC) methodology

Research Method · All research methods
Evidence · 6 claims (5 for, 1 mixed) · 5 studies (3 causal, 1 quant-synthesis, 1 theoretical), q2–q3 · 1 of 5 report an effect size · 6 claims rest on one study

Description

A within-study comparison examines "whether and in what context nonexperimental methods can successfully replicate the results of randomized experiments." The article situates its contribution in this growing literature and applies a WSC to charter school effects on student achievement, using a randomized study with control-group crossover as the experimental benchmark.

Accounts

  • Within-study comparison (WSC) methodology: A within-study comparison examines "whether and in what context nonexperimental methods can successfully replicate the results of randomized experiments." The article situates its contribution in this growing literature and applies a WSC to charter school effects on student achievement, using a randomized study with control-group crossover as the experimental benchmark. (Brian Gill et al. (2016))
  • Within-study comparison of RD and RCT estimates at the shared cutoff as a test of RD validity: The article's methodological framework is the within-study comparison: each of 15 studies assesses "the correspondence between causal estimates from an RD study and a randomized control trial (RCT) when the estimates are made at the same cutoff point where they should not differ asymptotically." Because statistical error, imperfect design implementation, and many analysis options could still produce differences, the framework tests whether they do, "assuming that the bias potential is greater with RDs than RCTs." It is used both to test internal validity and, by examining variation in bias across studies, external validity. (Duncan D. Chaplin et al. (2018))
  • Match causal estimands when using within-study comparisons to validate nonexperimental methods: The article states, citing Cook, Shadish, and Wong (2008), that "WSCs require that the experimental and nonexperimental methods assess the same causal estimand." This principle guides the design of replication studies: before judging whether a nonexperimental method replicates an experimental result, researchers must ensure both approaches target the same quantity. The article operationalizes this by comparing to ITT estimates under noncompliance. (Brian Gill et al. (2016))
  • Matched randomized crossover design for within-student comparison of an instructional treatment: A course-level experimental pattern in which each student serves as their own control across units of a course. Students are assigned to treatment and control in different orders over the units (e.g., TCTC, CTCT, TCCT, CTTC), which, as the authors note, "eliminates what is arguably the largest source of variability in educational studies: variation between students." Matching students on covariates before assigning complementary treatments further guards against baseline differences. (Sun DL et al. (2015))
  • RCT-benchmark design using student-level charter school data with standardized achievement test outcomes: The study's design uses student-level data drawn from a large-scale randomized controlled trial of charter schools as a benchmark against which nonexperimental regression and propensity score estimators are tested for bias. The key outcome measures are standardized achievement tests. The article states it addresses prior studies' limitations by "using student-level data based on a large-scale RCT of charter schools for which standardized achievement tests are the key outcome measure." (Kenneth Fortson et al. (2015))
  • A new replication approach comparing nonexperimental estimates to experimental ITT estimates rather than CACE: The article develops a replication method for within-study comparisons in settings with control-group noncompliance. Instead of requiring the nonexperimental estimate to match the complier average causal effect, the approach uses "nonexperimental methods to produce an estimate that can be compared to the experimental intent-to-treat (ITT) impact estimate rather than the CACE." This aligns the causal estimands when some control-group members are always-takers who receive treatment anyway. (Brian Gill et al. (2016))
  • Benchmark nonexperimental methods against experimental ITT estimates, accounting for control crossover: When judging whether nonexperimental methods can recover causal impacts, the article recommends using the experimental ITT estimate as the standard of comparison. It notes that "Ideally, nonexperimental methods that aim to replicate the results of rigorous randomized experiments focus on the intent to treat (ITT) experimental impact estimate," and that the comparison must handle "substantial control crossover" in the experiment. (Brian Gill et al. (2013))
  • Synthetic within-study comparison design for assessing RD performance: A methodological artifact in which the authors "generate synthetic RD data sets from experimental data sets" from two recent evaluations of educational interventions—the Educational Technology Study and the Teach for America Study—and "compare the RD impact estimates to the experimental estimates of the same intervention." This lets RD performance be benchmarked against randomized estimates for the same interventions. The article uses it both to test well-implemented RD and to study manipulation bias. (Philip Gleason et al. (2018))
  • Measurement local optima: canonical executive function tasks may reflect measurement inertia rather than genuine invariance: The article argues that operationalizations of psychological constructs settle into "local optima": "definitions that are reliable, interpretable, and widely accepted, yet not necessarily epistemically optimal for addressing all theoretical questions." Canonical tasks like the Flanker and Stroop are optimized for reliability, comparability, and stable individual differences, which can reduce sensitivity to transient, state-based perturbations. Apparent stability under perturbation may therefore reflect properties of the measurement instrument rather than genuine invariance of the underlying process. (David Moreau (2026))
  • Parallel-groups comparison of matched class sections as a method for investigating questions of teaching method: The article pilots a classroom-experiment design in which two sections of the same course, matched on sex, age, environment and intended occupation, are taught by different approaches with the same teacher, laboratory, texts and examinations. The author writes that "[t]he method of conducting the experiment, by the use of parallel groups ... is comparatively new, and should prove of service in questions of method of teaching", and the summary judges it "promises to yield valuable results" if refined through repeated tests. (Gilbert (1909))

Claims

-

Key Sources

  • Brian Gill, Joshua Furgeson, Hanley Chiang, Bing-Ru Teh, Joshua Haimson, Natalya Verbitsky-Savitz. (2016). Replicating Experimental Impact Estimates With Nonexperimental Methods in the Context of Control-Group Noncompliance. Statistics and Public Policy, vol. 3, issue 1. https://www.mathematica.org/publications/2015-replicating-experimental-impact-estimates-with-nonexperimental-methods-in-the-context
  • Duncan D. Chaplin, Thomas D. Cook, Jelena Zurovac, Jared S. Coopersmith, Mariel M. Finucane, Lauren N. Vollmer, Rebecca E. Morris. (2018). The Internal and External Validity of the Regression Discontinuity Design: A Meta-Analysis of 15 Within-Study Comparisons. Journal of Policy Analysis and Management, vol. 37, issue 2. https://www.mathematica.org/publications/the-internal-and-external-validity-of-the-regression-discontinuity-design-a-meta-analysis-of-15
  • Kenneth Fortson, Philip Gleason, Emma Kopa, Natalya Verbitsky-Savitz. (2015). Horseshoes, Hand Grenades, and Treatment Effects? Reassessing Whether Nonexperimental Estimators are Biased. Economics of Education Review, vol. 44. https://www.mathematica.org/publications/horseshoes-hand-grenades-and-treatment-effects-reassessing-whether-nonexperimental-estimators
  • Philip Gleason, Alexandra Resch, Jillian Berk. (2018). RD or Not RD: Using Experimental Studies to Assess the Performance of the Regression Discontinuity Approach. Evaluation Review, vol. 42. https://www.mathematica.org/publications/rd-or-not-rd-using-experimental-studies-to-assess-the-performance-of-the-regression-discontinuity
Page metadata
Field Value
type research-method
id within-study-comparison-wsc-methodology
aliases rct-benchmark-design-for-validating-nonexperimental-charter-school-estimators, replication-approach-comparing-nonexperimental-estimates-to-experimental-itt-estimates-rat, synthetic-within-study-comparison-design-for-assessing-regression-discontinuity-performanc
title Within-study comparison (WSC) methodology
description A within-study comparison examines "whether and in what context nonexperimental methods can successfully replicate the results of randomized experiments." The article situates its contribution in this growing literature and applies a WSC to charter school effects on student achievement, using a rand
status review
moved_from element/within-study-comparison-wsc-methodology, theory/within-study-comparison-rd-rct-cutoff
generated by process:sweep-kinds on 2026-10-10
trust unverified — no one has confirmed this page's content
sources 4 — brian-gill-2016, duncan-d-chaplin-2018, kenneth-fortson-2015, philip-gleason-2018

Frontmatter is the page's machine-readable record — see the Schema & Guide.