In-game behaviors collected unobtrusively in serious games show promising relationships with candidate ability level in the target professional competency¶

Claim · All claims
Evidence · 1 study · review r3 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Reviewed serious games used accuracy, time spent, and log data as unobtrusive measures, and in-game measures were found to have promising relationships with competency levels. → Bijl 2024

Evidence¶

Bijl 2024¶

Bijl, A., Veldkamp, B. P., Wools, S., & de Klerk, S. (2024). Serious games in high-stakes assessment contexts: a systematic literature review into the game design principles for valid game-based performance assessment. Education Tech Research Dev, 72, 2041–2064. https://doi.org/10.1007/s11423-024-10362-0

q2 · i? · review · r3

The review's unobtrusive data collection synthesis (Table 6) identifies accuracy measures in multiple games, completion-time measures (e.g., Wii Laparoscopy correlating with a validated laparoscopic skills measure), and log data in one cognitive-skills game. It quotes that in-game measures had "promising relationships with competency levels".

"Finally, the use of log data was found in one SG targeted at cognitive skills (Steinrücke et al., 2020). In the Dilemma Game, in-game measures collected during gameplay were found to have promising relationships with competency levels."

Discussion¶

Related Claims¶

• Adaptivity is rarely implemented in serious games for professional competencies, no reviewed game adjusts difficulty down based on player actions, and none were found for cognitive skills — related
• Relative-Hards Solved Complex, a log measure of using complex code on personally difficult levels, predicts time spent on an unsolvable challenge task — a narrower finding that bears on this claim
• In-game willingness to solve hard levels with complex code is positively associated with posttest coding knowledge, controlling for pretest — a narrower finding that bears on this claim
• Automated detectors of disengaged behaviors inferred from interaction-log data alone reached AUC over 0.8 for gaming and correlated 0.55 with observed off-task behavior — related
• Serious games can provide an authentic physical context and task for professional competencies, though a small number of successful games used neither — related
• Serious games are frequently successful in teaching, training, or assessing professional competencies, with reported success rates of 84% for cognitive skills, 83% for knowledge, and 100% for motor skills — related