Conditioning results from multiple disengagement detection methods on response times shows how the conditional approach aids threshold setting and misclassification avoidance¶

Claim · All claims
Evidence · 1 study · associational r? · q2
Currency · newest study 2018 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2021: tool and model accuracy changes quickly

Subclaims¶

q2 i? The study conditions results from a variety of detection methods used to identify disengaged survey responses on response times. → Soland 2018 q2 i? The conditional approach may be useful in identifying where to set response time thresholds for survey items and in avoiding misclassification when using other detection methods. → Soland 2018

Evidence¶

Soland 2018¶

Soland, J., Wise, S., & Gao, L. (2018). Identifying disengaged survey responses: New evidence using response time metadata. Applied Measurement in Education, 32(2), 151-165. https://www.nwea.org/research/publication/identifying-disengaged-survey-responses-new-evidence-using-response-time-metadata/ doi:10.1080/08957347.2019.1577244

q2 · i? · associational · r?

The abstract describes the study's method and use case: results from "a variety of detection methods" are conditioned on response times, showing how the approach may help set response time thresholds for survey items and avoid misclassification. No effect sizes or statistics are printed in the supplied text.

"In this study, we condition results from a variety of detection methods used to identify disengaged survey responses on response times. We then show how this conditional approach may be useful in identifying where to set response time thresholds for survey items, as well as in avoiding misclassification when using other detection methods."

Discussion¶

Related Claims¶

• Validating decision rules for detecting disengaged survey responses is problematic because surveys have no correct answers — related
• Response-time threshold-setting methods for detecting noneffortful item responses were compared using reading scores from over 728,923 US students in 2,056 schools — related
• Teachers report difficulty locating conceptual gaps because students stop progressing silently after difficulty, making disengagement hard to detect during class — related
• Threshold-setting methods are compared using reading scores from over 728,923 students on a computer-adaptive test of nearly 12,000 items — related
• Disengaged responding biases observed scores from achievement tests and surveys in practically and statistically significant ways — related