Interpreting differences in school-aggregate test scores as real differences in school performance raises accuracy concerns even under routine schooling conditions, which the pandemic may exacerbate¶

Claim · All claims
Evidence · 1 study · associational r? · q1
Currency · newest study 2022 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2025: tool and model accuracy changes quickly

Subclaims¶

q1 i? There are serious concerns about the accuracy of interpreting differences in aggregate scores as real and meaningful differences in school progress and performance, even under routine conditions. → Jonathan Schweig 2022

Evidence¶

Jonathan Schweig 2022¶

Jonathan Schweig, Megan Kuhfeld, Andrew McEachin, Melissa Diliberti, Louis Mariano. (2022). Changes in school composition during the COVID-19 pandemic: Implications for school-average interim test score use. NWEA Research. https://www.nwea.org/research/publication/changes-in-school-composition-during-the-covid-19-pandemic-implications-for-school-average-interim-test-score-use/ doi:10.7249/rra1037-2

q1 · i? · associational · r?

The article's stated rationale: reliance on the assumption that aggregate-score differences represent real differences in school progress, with accuracy concerns the authors say the pandemic may exacerbate. The supplied text offers no effect size or test statistic for this assertion.

"There are serious concerns about the accuracy of such interpretations even under routine schooling conditions, but the pandemic may exacerbate these issues and further compromise the comparability of these test scores."

Discussion¶

Related Claims¶

• School officials regularly use school-aggregate test scores to monitor school performance and make policy decisions — related
• No-credit rates during the pandemic varied considerably across elementary schools, including among schools serving similar student populations — a narrower finding that bears on this claim
• Changes in school composition over time during the COVID-19 pandemic may contaminate school-aggregate test scores and produce faulty comparisons with historical and proximal aggregate scores — a narrower finding that bears on this claim
• An argument-based approach to test score effectiveness can be illustrated by contrasting effectiveness arguments for interim assessments based on a conventional test blueprint versus one augmented with learning progressions — related
• School-level aggregated scores on the computer-based PISA-Based Test for Schools were highly stable despite considerable differential engagement among schools — related
• Disengaged test taking in international assessment programs threatens the validity of inferences about a country's educational attainment — related
• Officials raised equity concerns that closing low-performing charter schools may not better serve students when alternative district schools perform even worse — related
• The article argues that interpreting the meaning and significance of presented data is the important problem, and that current emphasis on reporting conditions minimizes interpretation — related
• A test score represents a pupil's performance under given conditions, not the pupil's ability, and conditions vary with the examiner and the pupil. — related