Provide assessment developers and administrators guidance on tradeoffs of threshold-setting methods for identifying noneffortful responses¶

Strategy · All strategies
Evidence · 5 claims (5 for) · 2 studies (1 associational, 1 design), q1–q3 · 0 of 2 report an effect size · 5 claims rest on one study

Description¶

The article's stated purpose is to "help provide guidance to developers and administrators of large-scale assessments on the tradeoffs involved in using a given method to identify noneffortful responses." It pursues this by comparing multiple response time threshold-setting methods on large-scale computer-adaptive test data. The guidance concerns selecting among detection methods rather than instructing learners directly.

Design Implications¶

Context¶

Requirements¶

• Response time data must be available from a large-scale assessment to apply the threshold-setting methods compared

Constraints¶

• The guidance is developed from a reading CAT in grades 3–8 in the United States, per the abstract

Target Learners¶

• 3rd–8th grade students taking large-scale interim assessments

Target Learning Goals¶

• valid interpretation of achievement test scores by detecting noneffortful responses

Claims¶

• Threshold-setting methods are compared using reading scores from over 728,923 students on a computer-adaptive test of nearly 12,000 items [+W]
• Prior threshold-method comparison studies typically used simulated or small-scale data, or non-CAT large-scale data with few items [+W]
• Detecting noneffortful responses requires setting response time thresholds that separate effortful from noneffortful responses [+W]
• Response-time threshold-setting methods for detecting noneffortful item responses were compared using reading scores from over 728,923 US students in 2,056 schools [+W]
• Information-based rapid-guessing threshold methods set thresholds more reliably than methods based on response accuracy or visual inspection [+W]

Related Strategies¶

Examples¶

-

Key Sources¶

• James Soland, Megan Kuhfeld, Joseph Rios. (2021). Comparing different response time threshold setting methods to detect low effort on a large-scale assessment. Large-scale Assessments in Education. https://doi.org/10.1186/s40536-021-00100-w