Estimate cumulative impacts of district-wide reforms by comparing achievement changes to similar students in similar geographic areas without the reforms¶

Strategy · All strategies
Evidence · 1 claim (1 for) · 1 study (1 causal), q2 · 0 of 1 report an effect size · 1 claim rests on one study

Description¶

The study estimated cumulative impacts of district-wide reforms through a comparative approach: it analyzed "how changes in achievement levels of DC schools compare to changes observed for similar students in similar geographic areas without such reforms." This comparison strategy allows reform-period achievement changes to be benchmarked against what would plausibly have occurred without the reforms. The brief presents this as the method underlying its impact estimates.

Design Implications¶

Context¶

Requirements¶

• Achievement data for the reform district and for comparable students and geographic areas without the reforms

Constraints¶

• Estimates are associations relative to comparison areas, not causal effects of specific reform components

Target Learners¶

• K-12 students in public schools in a reforming district

Target Learning Goals¶

• District-wide student achievement in math and reading as measured by NAEP

Claims¶

• The DC reform analysis used advanced statistical methods to account for pre-reform differences between DC and other areas and general trends afterward [+W]

Related Strategies¶

• Evaluate charter school impacts by matching students to demographically and academically similar comparison students
• Evaluate principal replacement by comparing achievement changes in replaced-principal schools to matched comparison schools that kept the same principal

Examples¶

-

Key Sources¶

• Dallas Dotter, Duncan Chaplin, and Maria Bartlett. (2021). Measuring the Impacts of School Reforms in the District of Columbia on Student Achievement (Issue Brief). Mathematica. https://www.mathematica.org/publications/measuring-the-impacts-of-school-reforms-in-the-district-of-columbia-on-student-achievement