Prior threshold-method comparison studies typically used simulated or small-scale data, or non-CAT large-scale data with few items¶

Claim · All claims
Evidence · 1 study · design r2 · q1
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q1 i? Previous studies comparing threshold-setting methods typically used simulated or small-scale data, or large-scale data that was not from a CAT, used few items, or did not examine many methods. → James Soland 2021

Evidence¶

James Soland 2021¶

James Soland, Megan Kuhfeld, Joseph Rios. (2021). Comparing different response time threshold setting methods to detect low effort on a large-scale assessment. Large-scale Assessments in Education. https://doi.org/10.1186/s40536-021-00100-w

q1 · i? · design · r2

The abstract characterizes the research gap: prior comparisons "typically do so using simulated or small-scale data", and large-scale studies often were not from a CAT, used only a handful of items, or examined methods incomprehensively.

"While other studies have compared the efficacy of different threshold-setting methods, they typically do so using simulated or small-scale data."

Discussion¶

Related Claims¶

• Threshold-setting methods are compared using reading scores from over 728,923 students on a computer-adaptive test of nearly 12,000 items — a narrower finding that bears on this claim
• Response-time threshold-setting methods for detecting noneffortful item responses were compared using reading scores from over 728,923 US students in 2,056 schools — related
• Detecting noneffortful responses requires setting response time thresholds that separate effortful from noneffortful responses — related
• Information-based rapid-guessing threshold methods set thresholds more reliably than methods based on response accuracy or visual inspection — related
• The general effort-measurement process is illustrated with data from a large-scale assessment program