ISAT scores are not comparable over time: students with identical earlier and later test scores earned different seventh-grade ISAT scores depending on the year tested¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r2–r3 · q2
Currency · newest study 2011 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2014: tool and model accuracy changes quickly

Subclaims¶

q2 i? Among students scoring 15 on the EXPLORE exam in ninth grade, average seventh-grade ISAT scores rose from 247.5 in 2006 to 251 in 2007 and 254 in 2008 despite nearly identical fourth-grade ITBS scores. → Luppescu 2011 q2 i? The number of correct answers required to meet ISAT math standards declined across years, and similar cohort analyses for other cohorts and for reading found similar results. → Luppescu 2011 (2)

Evidence¶

Luppescu 2011¶

Luppescu, S., Allensworth, E. M., Moore, P., de la Torre, M., Murphy, J., with Jagesic, S. (2011). Trends in Chicago's Schools Across Three Eras of Reform. Consortium on Chicago School Research, University of Chicago. https://consortium.uchicago.edu/publications/trends-chicagos-schools-across-three-eras-reform-full-report

q2 · i? · associational · r2

Cohort-comparison analysis matching students on ninth-grade EXPLORE scores (15) and comparing their seventh-grade ISAT and fourth-grade ITBS scores, shown in Figure 2. Average seventh-grade ISAT scores differed by year (247.5, 251, 254) despite nearly identical ITBS scores, suggesting ISAT scores are not comparable over time. No effect size is printed.

"The average for these students in 2006 was 247.5; in 2007 it was 251; and in 2008 it was 254."

Luppescu 2011 (2)¶

Luppescu, S., Allensworth, E. M., Moore, P., de la Torre, M., Murphy, J., with Jagesic, S. (2011). Trends in Chicago's Schools Across Three Eras of Reform. Consortium on Chicago School Research, University of Chicago. https://consortium.uchicago.edu/publications/trends-chicagos-schools-across-three-eras-reform-full-report

q2 · i? · associational · r3

Documented analysis of ISAT cut-score changes in Chapter 2: the correct-answer threshold for meeting fifth-grade math standards fell from 36 (2006) to 35 (2008) and further to 33 and 32 in 2009 and 2010. The report also replicated the cohort analysis for reading and other cohorts. No effect size is printed.

"in 2006 a fifth grade student who answered 36 questions correctly on the math exam would be judged as meeting standards. In 2008, a student only needed to answer 35 answers correctly to meet standards."

Discussion¶

Related Claims¶

• Elementary/middle-grade math scores improved incrementally across all three eras while reading scores remained fairly flat for two decades — related
• Ninth-grade GPA predicts beginning-of-tenth-grade PLAN scores even after controlling for eighth-grade test scores, suggesting grades reflect new learning — related
• Percent meeting standards is an imprecise metric whose cut-score concentration produced the Era 2 misconception that Chicago taught basic skills better than high-level skills — related
• Most RSF-supported Cohort 1 schools increased the percentage of students meeting or exceeding ISAT standards from year 1 to year 2 — related
• In fifth-grade reading, ITBS scores declined significantly more than ISAT scores from 1999 to 2002 — related
• Districts' most frequent suggestions for improving ISAT are stability and consistency and faster turnaround, each urged by about one-quarter of districts — related
• ITBS and ISAT inclusion rules differ: ISAT reports scores for all test-takers, while about 20 percent of ITBS test-takers are excluded from public reporting — related
• Test preparation demands slow instruction, with the spring review season beginning earlier and lasting longer each year of the study — related
• ISAT cut scores differ by grade and subject, with eighth-grade mathematics requiring a high score and eighth-grade reading a relatively low one to meet standards — related
• After two years, RSF Cohort 1 students generally performed at the same levels as propensity-score-matched comparison students, with few statistically significant differences — related