Differences in rapid guessing between subgroups generally do not impact substantive interpretations of achievement gaps¶

Claim · All claims
Evidence · 1 study · associational r? · q3
Currency · newest study 2018 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q3 i? Achievement gap estimates were estimated unconditional and conditional on measures of rapid guessing, and substantive interpretations of gaps were generally unchanged. → James Soland 2018

Evidence¶

James Soland 2018¶

James Soland. (2018). Are achievement gap estimates biased by differential student test effort? Putting an important policy metric to the test. Teachers College Record, 120(12), 1-26. https://www.nwea.org/research/publication/are-achievement-gap-estimates-biased-by-differential-student-test-effort-putting-an-important-policy-metric-to-the-test/ doi:10.1177/016146811812001202

q3 · i? · associational · r?

Sensitivity analysis comparing achievement gap estimates unconditional and conditional on measures of rapid guessing. The study reports these differences "generally do not impact substantive interpretations of achievement gaps." No effect sizes or test statistics are printed in the available text.

"However, these differences in rapid guessing generally do not impact substantive interpretations of achievement gaps, though basic conclusions about male-female gaps and changes in gaps as students progress through school may change when models account for test effort."

Discussion¶

Related Claims¶

• Differential student test effort may bias achievement gap estimates — reports the opposite
• Basic conclusions about male-female achievement gaps and changes in gaps as students progress through school may change when models account for test effort — reports the opposite
• Test effort differs substantially by student subgroup, with males rapidly guessing nearly twice as often as females in later grades and Black students rapidly guessing more often than White students — related
• Rapid guessing did not appear to substantively affect teacher value-added estimates under the value-added specifications used — related
• Rapid-guessing behavior can distort test scores and adversely affect measurement — a broader claim this one bears on
• Rapid guessing had mostly minimal effects on estimates of school contributions to student growth in this sample — related
• Disengaged test-taking rates varied greatly by subgroup, grade, and term in this sample — related
• The article investigates whether achievement gap estimates are sensitive to test motivation, framing the question as achievement gap versus engagement gap — reports the opposite
• Rapid guessing can be examined over three years using six test administrations to distinguish trait-like from situational disengagement — related
• Accounting for test disengagement had minimal impact on estimates of school growth contributions, achievement gaps, and summer learning loss — a broader claim this one bears on
• Motivation filtering reversed the estimated Black-White fall-to-fall growth gap direction in later grades — reports the opposite
• Disengaged responding biases observed scores from achievement tests and surveys in practically and statistically significant ways — reports the opposite