Disengaged responding biases observed scores from achievement tests and surveys in practically and statistically significant ways¶

Claim · All claims
Evidence · 1 study · associational r? · q2
Currency · newest study 2018 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2021: tool and model accuracy changes quickly

Subclaims¶

q2 i? Disengaged responding is a phenomenon that often biases observed scores from achievement tests and surveys in practically and statistically significant ways. → Soland 2018

Evidence¶

Soland 2018¶

Soland, J., Wise, S., & Gao, L. (2018). Identifying disengaged survey responses: New evidence using response time metadata. Applied Measurement in Education, 32(2), 151-165. https://www.nwea.org/research/publication/identifying-disengaged-survey-responses-new-evidence-using-response-time-metadata/ doi:10.1080/08957347.2019.1577244

q2 · i? · associational · r?

The article's abstract asserts that disengaged responding "often biases observed scores from achievement tests and surveys in practically and statistically significant ways." No effect size or test statistic is printed in the supplied text.

"Disengaged responding is a phenomenon that often biases observed scores from achievement tests and surveys in practically and statistically significant ways."

Discussion¶

Related Claims¶

• Test disengagement could undermine inferences drawn from observed achievement scores — a narrower finding that bears on this claim
• Filtered students had much lower effort-moderated achievement, suggesting motivation filtering may bias mean estimates — related
• Validating decision rules for detecting disengaged survey responses is problematic because surveys have no correct answers — related
• Content alignment between a test and its content domain represents key evidence for validating test score inferences — related
• Conditioning results from multiple disengagement detection methods on response times shows how the conditional approach aids threshold setting and misclassification avoidance — related
• One measurement article argues that low test-taking motivation biases an individual student's achievement test score downward — a narrower finding that bears on this claim
• Differential rapid guessing often resulted in test events with meaningfully distorted item content representation — a narrower finding that bears on this claim
• Rapid-guessing behavior can distort test scores and adversely affect measurement — a narrower finding that bears on this claim
• Test effort differs substantially by student subgroup, with males rapidly guessing nearly twice as often as females in later grades and Black students rapidly guessing more often than White students — related
• Differences in rapid guessing between subgroups generally do not impact substantive interpretations of achievement gaps — reports the opposite