Evaluate charter school impacts by matching students to demographically and academically similar comparison students¶

Strategy · All strategies
Evidence · 4 claims (3 for, 1 mixed) · 4 studies (2 causal, 2 associational), q2–q3 · 0 of 4 report an effect size · 4 claims rest on one study

Description¶

The report measures a charter school's impact by identifying a comparison group of students in other district and charter schools who were similar on observed characteristics, then comparing outcomes. As the report states, "we identified a group of students in other Kansas City district and charter schools who had similar demographic characteristics and achievement at the end of 4th grade." This matching strategy grounds the year-7 impact estimates.

Design Implications¶

Context¶

Requirements¶

• Baseline data on student demographics and prior achievement (end of 4th grade) for both groups
• Outcome data for students in grades 5 through 11 in both the treated and comparison groups

Constraints¶

• Causal estimates depend on the quality of the match; ACT impact could not be calculated because necessary data were not available

Target Learners¶

• Students in grades 5 through 11 in urban charter and district schools

Target Learning Goals¶

• Estimating school-level impacts on achievement growth and college-readiness outcomes

Claims¶

• Charter high school students had better attendance and test scores than comparable non-charter students, controlling for incoming characteristics [+W]
• Charter school students show more annual learning growth in reading and math than matched traditional public school peers [+W]
• Pennsylvania charter school students show similar reading growth but weaker math growth (about 30 fewer days of learning) than matched TPS peers [~W]
• In the full-sample analysis, baseline demographic and prior-achievement differences between treatment and matched comparison groups were minimal, with effect sizes below the WWC threshold [+W]

Related Strategies¶

• Use a matched comparison group design with qualitative mechanism exploration to evaluate charter school impacts
• Estimate cumulative impacts of district-wide reforms by comparing achievement changes to similar students in similar geographic areas without the reforms
• Use community-area demographic descriptor data as an input to school planning and reform

Examples¶

-

Key Sources¶

• Matthew Johnson, Alicia Demers. (2019). Ewing Marion Kauffman School Year 7 Impacts. Oakland, CA: Mathematica. https://www.mathematica.org/publications/ewing-marion-kauffman-school-year-7-impacts