Response-time threshold-setting methods for detecting noneffortful item responses were compared using reading scores from over 728,923 US students in 2,056 schools¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2021 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · overdue for a recheck since 2024: tool and model accuracy changes quickly

Subclaims¶

q2 i? A large-scale study compared threshold-setting methods for detecting noneffortful item responses using reading test scores from over 728,923 3rd-8th-grade students in 2,056 schools in one U.S. state (the abstract says "across the United States", but the Methods section places the data in one state). → Soland 2021

Evidence¶

Soland 2021¶

Soland, J., Kuhfeld, M., & Rios, J. (2021). Comparing different response time threshold setting methods to detect low effort on a large-scale assessment. Large-scale Assessments in Education, 9, 8. doi:10.1186/s40536-021-00100-w

q2 · i? · associational · r2

An NWEA publications listing (read in place of the article) gives this study's summary: a large-scale comparison of threshold-setting methods on reading test scores, covering "over 728,923 3rd–8th-grade students in 2,056 schools across the US". No per-method results are printed in the listing. The article's Methods section states that the data are from one U.S. state, so "across the US" describes the summary's wording, not a nationwide sample.

"This study uses reading test scores from over 728,923 3rd–8th-grade students in 2,056 schools across the US to compare threshold-setting methods to detect noneffortful item responses."

Discussion¶

Related Claims¶

• Threshold-setting methods are compared using reading scores from over 728,923 students on a computer-adaptive test of nearly 12,000 items — possibly the same claim (merge candidate)
• Prior threshold-method comparison studies typically used simulated or small-scale data, or non-CAT large-scale data with few items — related
• Detecting noneffortful responses requires setting response time thresholds that separate effortful from noneffortful responses — related
• Low examinee effort is a major threat to valid uses of many test scores — a broader claim this one bears on
• CATs can have important advantages over conventional tests in identifying instances when examinees exhibit low effort — related
• Conditioning results from multiple disengagement detection methods on response times shows how the conditional approach aids threshold setting and misclassification avoidance — related
• Information-based rapid-guessing threshold methods set thresholds more reliably than methods based on response accuracy or visual inspection — related
• Unmotivated examinees present a major threat to the validity of scores in low-stakes testing programs — related
• The general effort-measurement process is illustrated with data from a large-scale assessment program — possibly the same claim (merge candidate)