Publicly reported school accountability statistics are not useful for measuring achievement trends over time¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2–r3 · q2
Currency · newest study 2011 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2014: tool and model accuracy changes quickly

Subclaims¶

q2 i? Changes in tests, standards, scoring, test administration, exclusion policies, and student composition make publicly reported CPS statistics non-comparable across years. → Luppescu 2011 q2 i? Publicly reported data suggested tremendous elementary progress while the report's adjusted analysis showed only incremental math gains and almost no reading growth. → Luppescu 2011 (2)

Evidence¶

Luppescu 2011¶

Luppescu, S., Allensworth, E. M., Moore, P., de la Torre, M., Murphy, J., with Jagesic, S. (2011). Trends in Chicago's Schools Across Three Eras of Reform. Consortium on Chicago School Research, University of Chicago. https://consortium.uchicago.edu/publications/trends-chicagos-schools-across-three-eras-reform-full-report

q2 · i? · associational · r3

Methodological analysis in Chapter 2 identifying five issues that make public statistics non-comparable: test and scoring changes, the imprecise percent-meeting-standards metric, promotion-policy effects on grade composition, changing exclusion rates, and demographic change. No effect size is printed.

"the publicly reported statistics that are used by CPS and other school districts to gauge progress are simply not useful for measuring trends over time"

Luppescu 2011 (2)¶

Luppescu, S., Allensworth, E. M., Moore, P., de la Torre, M., Murphy, J., with Jagesic, S. (2011). Trends in Chicago's Schools Across Three Eras of Reform. Consortium on Chicago School Research, University of Chicago. https://consortium.uchicago.edu/publications/trends-chicagos-schools-across-three-eras-reform-full-report

q2 · i? · associational · r2

Comparison of publicly reported proficiency trends with the report's own student-level rescaled trends, from the executive summary. The discrepancy is attributed to changes in test content and scoring that make year-over-year comparisons nearly impossible without complex statistical adjustment. No effect size is printed.

"publicly reported statistics indicate that CPS has made tremendous progress in elementary math and reading tests, while this analysis demonstrates only incremental gains in math and almost no growth in reading"

Discussion¶

Related Claims¶

• Elementary/middle-grade math scores improved incrementally across all three eras while reading scores remained fairly flat for two decades — related
• Racial achievement gaps in Chicago increased in all three eras, with African American students falling behind all other groups, especially in Era 3 — related
• Percent meeting standards is an imprecise metric whose cut-score concentration produced the Era 2 misconception that Chicago taught basic skills better than high-level skills — a narrower finding that bears on this claim
• Graduates' academic qualifications improved alongside graduation rates, with ACT scores rising from 16.7 to 18.6 between 2003 and 2014 — reports the opposite
• Changes in student backgrounds and incoming achievement explain graduation improvements before 2006 but not after, because entering test scores stopped improving and declined — related
• CPS graduation rates increased by 22.4 percentage points over 16 years, from 52.4 percent among students who turned 19 in 1998 to 74.8 percent in 2014 — related
• Ninth-grade GPAs rose steadily in CPS from 2006 to 2013 alongside improvements in test scores, attendance, and behavior — reports the opposite
• Profile trends labeled up or down are determined by statistical criteria, so those labels denote statistically significant trends — related
• Increased score variability was primarily driven by spreading out at the bottom end of the achievement distribution — related
• Rising CPS graduation rates have been accompanied by higher achievement, including ACT and GPA gains — related
• Data coding issues could account for some graduation-rate improvement between 2005 and 2008, but not for improvements in subsequent years — related
• ITBS and ISAT inclusion rules differ: ISAT reports scores for all test-takers, while about 20 percent of ITBS test-takers are excluded from public reporting — a narrower finding that bears on this claim