Validating decision rules for detecting disengaged survey responses is problematic because surveys have no correct answers¶

Claim · All claims
Evidence · 1 study · associational r? · q2
Currency · newest study 2018 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2021: tool and model accuracy changes quickly

Subclaims¶

q2 i? Unlike on achievement tests, surveys have no correct answers, so validating decision rules for detecting disengaged responses is problematic. → Soland 2018

Evidence¶

Soland 2018¶

Soland, J., Wise, S., & Gao, L. (2018). Identifying disengaged survey responses: New evidence using response time metadata. Applied Measurement in Education, 32(2), 151-165. https://www.nwea.org/research/publication/identifying-disengaged-survey-responses-new-evidence-using-response-time-metadata/ doi:10.1080/08957347.2019.1577244

q2 · i? · associational · r?

The abstract states a validation problem: "unlike on achievement tests, there are no correct answers," and consequently "validating decision rules for detection methods is problematic." This is the study's stated rationale rather than a tested result.

"One major disadvantage when trying to detect disengaged responses on surveys is that, unlike on achievement tests, there are no correct answers. As a result, validating decision rules for detection methods is problematic."

Discussion¶

Related Claims¶

• Conditioning results from multiple disengagement detection methods on response times shows how the conditional approach aids threshold setting and misclassification avoidance — related
• Disengaged responding biases observed scores from achievement tests and surveys in practically and statistically significant ways — related