Effort monitoring with proctor notification feature for computer-based tests¶

Element · All elements
Evidence · 6 claims (6 for) · 4 studies (2 causal, 1 associational, 1 design), q2–q3 · 0 of 4 report an effect size · 6 claims rest on one study

Description¶

A computer-based testing feature that detects rapid guessing as it occurs and notifies proctors that a test taker has become disengaged. The article describes it as "a new effort monitoring feature that can detect rapid guessing, as it occurs, and notify proctors that a test taker has become disengaged." The feature was evaluated empirically for its effects on engagement, performance, and score validity.

Design Implications¶

Context¶

Requirements¶

• Requires computer-based test delivery so that rapid-guessing behavior can be detected in real time, and proctors available to receive and act on notifications.

Constraints¶

• The article evaluates it in the context of tests perceived as low stakes, where disengaged test taking occurs.

Target Learners¶

• K-12 students taking computer-based achievement tests

Target Learning Goals¶

• Valid measurement of what test takers know and can do

Claims¶

• Proctor notification after rapid-guessing detection improved test performance and yielded higher convergent validation evidence [+W]
• After a proctor notification triggered by rapid-guessing detection, test-taking engagement tended to increase [+W]
• Rapid-guessing behavior on computer-based tests signals disengaged test taking that threatens score validity [+W]
• Proctor notification when students disengage makes a significant impact on student test engagement [+W]
• The assessment's engagement feature had a consistently positive impact under both in-person and remote proctoring [+W]
• The test platform detects rapid guessing in real time and attempts to re-engage disengaged students to protect score validity [+W]

Related Elements¶

• Proctor notification feature for detecting and responding to student disengagement

Examples¶

• Use automated disengagement detection with proctor notification during remote low-stakes testing
• Notify test proctors when students disengage so students give their best effort on testing day
• Proctor Intervention for Rapid Guessing

Key Sources¶

• Steven Wise, Megan Kuhfeld, James Soland. (2019). The effects of effort monitoring with proctor notification on test-taking engagement, test performance, and validity. Applied Measurement in Education, 32(2), 183-192. https://www.nwea.org/research/publication/the-effects-of-effort-monitoring-with-proctor-notification-on-test-taking-engagement-test-performance-and-validity/