RTE provides an assessment of test-taking effort down to the level of individual item responses¶

Claim · All claims
Evidence · 1 study · design r1 · q2
Currency · newest study 2017 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2020: tool and model accuracy changes quickly

Subclaims¶

q2 i? Response Time Effort yields effort assessments at the level of individual item responses, not just whole tests. → Steven Wise 2017

Evidence¶

Steven Wise 2017¶

Steven Wise, Lingyun Gao. (2017). A general approach to measuring test-taking effort on computer-based tests. Applied Measurement in Education, 30(4), 343–354. https://www.nwea.org/research/publication/a-general-approach-to-measuring-test-taking-effort-on-computer-based-tests/ doi:10.1080/08957347.2017.1353992

q2 · i? · design · r1

The article's abstract reports, citing prior work, that Response Time Effort (RTE) "has been shown to provide an assessment of effort down to the level of individual item responses." The article describes this as established rather than newly demonstrated in the current study.

"In particular, Response Time Effort (RTE) has been shown to provide an assessment of effort down to the level of individual item responses."

Discussion¶

Related Claims¶

• Computer-based testing's routine capture of item-level response times has driven increased interest in applications of response time to measuring ability and achievement — related
• A general process for measuring item-level effort extends effort measurement to omitted and constructed responses — related
• RTE is limited to selected-response items that must be answered before advancing — related
• Test-taking engagement plays an essential role in the pursuit of valid scores — a broader claim this one bears on
• Solution behavior accuracy does not decrease across item positions, contradicting decreasing effort models of unmotivated test taking — related
• Test-taking disengagement distorts both individual and aggregated test scores — related
• In simulation, effort-moderated scoring (EM-CAT and EG-CAT) yields negligible bias and nominal 95% confidence interval coverage, unlike MLE-CAT scoring — related
• Effort-moderated scoring yields more accurate achievement scores and better person fit than traditional MLE scoring when rapid guessing is present — related
• An effort-guided CAT that holds the provisional achievement estimate constant after rapid guesses reduces standard errors and ameliorates severe item mis-targeting — related
• Responses used in effort-moderated scoring can sometimes reflect less-than-full, or partial, engagement, making E-M scores less trustworthy — related
• The general effort-measurement process is illustrated with data from a large-scale assessment program
• Very short response times (rapid guessing) indicate disengaged test taking in both high-stakes and low-stakes testing contexts — related