School-level aggregated scores on the computer-based PISA-Based Test for Schools were highly stable despite considerable differential engagement among schools¶

Claim · All claims
Evidence · 1 study · associational r? · q2
Currency · newest study 2019 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2022: tool and model accuracy changes quickly

Subclaims¶

q2 i? Although there was considerable differential engagement among schools taking the computer-based PISA-Based Test for Schools, aggregated school means were highly stable. → Steven Wise 2019

Evidence¶

Steven Wise 2019¶

Steven Wise, James Soland, Yuanchao Bo. (2019). The (non)impact of differential test taker engagement on aggregated scores. International Journal of Testing 20 (1), 55-77. https://www.nwea.org/research/publication/the-nonimpact-of-differential-test-taker-engagement-on-aggregated-scores/

q2 · i? · associational · r?

The study analyzed data from the computer-based version of the PISA-Based Test for Schools to examine distortive effects of differential engagement on aggregated school-level scores. The abstract reports that "the school means were highly stable due to two factors," despite "considerable differential engagement among schools."

"The results showed that, although there was considerable differential engagement among schools, the school means were highly stable due to two factors."

Discussion¶

Related Claims¶

• One measurement article argues that low test-taking motivation biases an individual student's achievement test score downward — the general claim this finding bears on, with the studies behind it
• Previous research found a meaningful correlation between countries' mean engagement and mean performance on large-scale international assessments, motivating validity questions about aggregated scores — related
• Test-taking disengagement distorts both individual and aggregated test scores — a broader claim this one bears on
• Two decades of research have produced advances in both the measurement of test-taking disengagement and understanding of its distortive impact on individual and aggregated scores — a broader claim this one bears on
• Distortive effects of disengagement on school means were diluted by the high proportion of students exhibiting no non-effortful behavior — a narrower finding that bears on this claim
• Interpreting differences in school-aggregate test scores as real differences in school performance raises accuracy concerns even under routine schooling conditions, which the pandemic may exacerbate — related
• Design effects from aggregation are tolerable with moderate numbers of groups and few covariates — related
• Disengagement produced both positive and negative distortion of individual student scores, which tended to cancel out much of the net distortive effect on school means — related