A response-time-conditional approach to detecting disengaged survey responses¶

Theory · All theories
Evidence · 3 claims (3 for) · 1 study (1 associational), q2 · 0 of 1 report an effect size · 3 claims rest on one study

Description¶

The article presents a conditional approach to identifying disengaged survey responses in which results from a variety of detection methods are conditioned on response times. Because "validating decision rules for detection methods is problematic" when there are no correct answers, the approach uses response time metadata as a common criterion against which detection methods can be evaluated. The article presents this conditional approach as useful for identifying where to set response time thresholds for survey items.

Design Implications¶

Context¶

Requirements¶

• Requires response time metadata collected alongside survey responses
• Requires results from detection methods that can be conditioned on response times

Constraints¶

• Applies to surveys, which unlike achievement tests have no correct answers against which to validate detection rules

Target Learners¶

• Test and survey respondents (K-12 and similar populations taking achievement tests and surveys)

Target Learning Objectives¶

• Accurate measurement of achievement and survey outcomes by detecting and correcting disengaged responses

Claims¶

• Conditioning Detection Methods On Response Times [+M]
• Disengaged responding biases observed scores from achievement tests and surveys in practically and statistically significant ways [+W]
• Validating decision rules for detecting disengaged survey responses is problematic because surveys have no correct answers [+W]

Related Theories¶

• Item response time metadata as a basis for identifying disengaged test responses
• Rapid-guessing detection framework using item response time thresholds

Examples¶

• Use response time to identify test taker disengagement as rapid-guessing behavior

Key Sources¶

• Soland, J., Wise, S., & Gao, L. (2018). Identifying disengaged survey responses: New evidence using response time metadata. Applied Measurement in Education, 32(2), 151-165. https://www.nwea.org/research/publication/identifying-disengaged-survey-responses-new-evidence-using-response-time-metadata/. doi:10.1080/08957347.2019.1577244