CREDO City Studies Project growth-benchmarking analysis approach¶

Element · All elements
Evidence · 9 claims (9 for) · 4 studies (2 causal, 2 associational), q2 · 0 of 4 report an effect size · 9 claims rest on one study

Description¶

The City Studies Project is a recurring reporting and analysis program that compares academic performance of student groups in selected U.S. cities to average statewide performance, "while accounting for differences in student demographic characteristics based on statistical analysis." It focuses on academic growth to isolate a school's impact on learning, compares charter, magnet, and innovation students to similar district students within each city, and translates effect sizes to days-of-learning using a benchmark (large >|0.2|, medium |0.05|-|0.2|, small <|0.05|) proposed by Kraft (2020). Cohort 1 covers 10 cities over the 2014-15 to 2018-19 growth periods.

Design Implications¶

Context¶

Requirements¶

• Statewide comparison data and student demographic characteristics used for statistical controls
• Test-taker data by city, sector, and subgroup (e.g., the 2018-19 snapshot in Table 1)

Constraints¶

• The report states objective interpretation and comparison of effect sizes across cities is difficult due to city- and state-specific macro- and micro-level factors
• Washington DC growth is benchmarked against the city average rather than the state average

Target Learners¶

• K-12 public school students in the 10 studied cities

Target Learning Goals¶

• Academic performance growth in reading and mathematics

Claims¶

• Charter school entrants have similar or lower achievement than incumbents in all studied cities, suggesting no cream skimming [+W]
• Charter schools in many of the 10 studied cities make overall positive contributions to students' learning growth relative to state averages [+W]
• Black, Hispanic, and poverty students enrolled in charter schools show stronger growth trends than the same subgroups citywide in several cities [+W]
• District schools in most studied cities show consistently weaker performance growth than state averages, with exceptions including Denver [+W]
• Innovation school students mostly do not deviate from citywide growth trends, with Denver innovation schools an outperforming exception [+W]
• Magnet schools across the studied cities show strong positive impact on students' performance growth compared to state averages [+W]
• CREDO's new race/ethnicity approach estimates full annual growth per group without benchmarking against any race/ethnicity [+W]
• Fort Worth charter school Black students outgrew Fort Worth district school Black students in reading, though the within-Fort Worth contrast was not statistically significant [+W]
• Charter school impacts differ sharply by locale: urban charters show stronger reading growth while suburban and town charters show weaker growth in both subjects [+W]

Related Elements¶

• Days of Learning translation of growth results

Examples¶

-

Key Sources¶

• Margaret E. Raymond, Won Fy Lee. (2022). City Studies Project Meta-Analysis by CREDO of Stanford University: Cohort 1 Cities. Center for Research on Education Outcomes, Stanford University. https://credo.stanford.edu/city-studies/