Higher tab accept rates are strongly associated with failing attention checks among CS1 students using an AI code completion tool¶

Claim · All claims
Evidence · 1 study · associational r1 · q2 · i3 large
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i3 Students with higher tab acceptance rates were far more likely to fail attention checks (rho = 0.73, p < 0.001). → Jessica Hutchison 2026

Evidence¶

Jessica Hutchison 2026¶

Jessica Hutchison, Ian Tyler Applebaum, Kenneth Angelikas, Kush Rakesh Patel, Phuoc Nguyen, Antonio Lazaro, Nicholas Rucinski, Rahad Arman Nabid, and Stephen MacNeil. (2026). To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks. Proceedings of the 31st ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE 2026). https://doi.org/10.1145/3803400.3809394

q2 · i3 · associational · r1

Spearman correlation analysis of session-level logs from a 60-minute in-person lab study in which 55 CS1 students completed the rainfall problem in Clover. The analysis found students with high tab accept rates "were far more likely to fail attention checks (𝜌= 0.73,𝑝< 0.001)", an effect size of rho = 0.73, and had lower average dwell time (rho = -0.47).

"Furthermore, students whoTAR were far more likely to fail attention checks (𝜌= 0.73,𝑝< 0.001). TARwas also strongly negatively correlated withADT, (𝜌=− 0.47)"

Discussion¶

Learner Variables¶

• Attention — outcome: instruction changes it

Related Claims¶

• Longer dwell time on AI suggestions is associated with better attention check performance but slightly lower task performance — related
• Log analysis reveals novel post-acceptance behaviors (slow tab accept, slow accept then modify) that show suggestion acceptance is a multi-step process — related
• Behavioral interaction metrics are weakly or inconsistently correlated with task performance in AI-assisted programming — related
• Mean struggle time before acceptance is 15.6% higher among true attrition users but does not reach statistical significance — related