Percent meeting standards is an imprecise metric whose cut-score concentration produced the Era 2 misconception that Chicago taught basic skills better than high-level skills¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2–r3 · q2
Currency · newest study 2011 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2014: tool and model accuracy changes quickly

Subclaims¶

q2 i? Year-to-year changes in percent meeting standards depend entirely on how many students score near the cut-off, so small score gains can look large and large gains can go unnoticed. → Luppescu 2011 q2 i? Growth was actually fairly equal among low- and high-achieving students in Era 2; the quartile-focus misread district progress because many students scored near the 25th percentile and few near the 75th. → Luppescu 2011 (2)

Evidence¶

Luppescu 2011¶

Luppescu, S., Allensworth, E. M., Moore, P., de la Torre, M., Murphy, J., with Jagesic, S. (2011). Trends in Chicago's Schools Across Three Eras of Reform. Consortium on Chicago School Research, University of Chicago. https://consortium.uchicago.edu/publications/trends-chicagos-schools-across-three-eras-reform-full-report

q2 · i? · associational · r3

Methodological argument in Chapter 2 (Issue 2) about dichotomous reporting metrics. The report also notes the wide "meets" band (e.g., grade-three reading 191–227, roughly the 50th to 90th percentile) has no singular meaning for later outcomes. No effect size is printed.

"Small improvements in test scores can result in many more students meeting norms/standards if there are many students close to the cut-off score, while large overall improvements in test scores can go unnoticed if there are few students with scores close to the cut-off."

Luppescu 2011 (2)¶

Luppescu, S., Allensworth, E. M., Moore, P., de la Torre, M., Murphy, J., with Jagesic, S. (2011). Trends in Chicago's Schools Across Three Eras of Reform. Consortium on Chicago School Research, University of Chicago. https://consortium.uchicago.edu/publications/trends-chicagos-schools-across-three-eras-reform-full-report

q2 · i? · associational · r2

Re-analysis of Era 2 score distributions reported in Chapter 2. The widespread belief that the district improved low achievers more than high achievers arose because many students scored near the 25th percentile and few near the 75th, so equal growth moved many across the lower cut-off. No effect size is printed.

"In fact, there was fairly equal growth among both low- and high-achieving students during the time period—the district was not better at educating low-skill than high-skill students."

Discussion¶

Related Claims¶

• Elementary/middle-grade math scores improved incrementally across all three eras while reading scores remained fairly flat for two decades — related
• Racial achievement gaps in Chicago increased in all three eras, with African American students falling behind all other groups, especially in Era 3 — related
• Publicly reported school accountability statistics are not useful for measuring achievement trends over time — a broader claim this one bears on
• ISAT scores are not comparable over time: students with identical earlier and later test scores earned different seventh-grade ISAT scores depending on the year tested — related
• The merit award program produced modest achievement improvements concentrated in schools close to the award threshold — related
• ISAT cut scores differ by grade and subject, with eighth-grade mathematics requiring a high score and eighth-grade reading a relatively low one to meet standards — related
• One-to-one correspondence between ITBS national quartiles and ISAT performance categories breaks down at the first and fourth quartiles — related
• Reading score distributions within A–F letter grades overlap heavily, so letter grades misrepresent the achievement of many individual students — related
• Reliability of state test scores is highest near cut-scores or the mean and much lower for very high- or low-performing students — related
• A mean-classification cut-score method outperformed a contrasting-groups-style mean method because it was more robust to outliers — related