Observable metrics computed from IDE telemetry improved held-out prediction of query imminence (AUROC .726) and guided versus dependent help-seeking (AUROC .717) over raw telemetry¶

Claim · All claims
Evidence · 2 studies · 1 associational r2, 1 design r3 · q2
Currency · no dated study recorded · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? For query-imminence prediction, observable metrics alone performed best at AUROC 0.726, an absolute improvement of 0.037 over raw telemetry (0.689). → pred-1 q2 i? For help-seeking-type prediction, observable metrics alone performed best at AUROC 0.717, an absolute improvement of 0.027 over raw telemetry (0.690). → pred-2

Evidence¶

pred-1¶

David Barron, Xiaohang Tang, Rezky Dwisantika, Minsun Kim, David H. Smith IV, Jiaming Cui, and Yan Chen. 2026. TutorTrace: A Dataset and Taxonomy for Classifying Learner Behavioral States during AI-Assisted Programming Education. https://arxiv.org/abs/2608.26184 doi:10.48550/arxiv.2608.26184

q2 · i? · design · r3

Cross-cohort Random Forest evaluation trained on Deployment 1 (n=190) and evaluated on held-out Deployment 2 (n=113), with semantic inputs excluded. "Observable metrics alone performed best at 0.726"; behavioral-sequence features (0.719) did not improve on observable metrics.

"For query-imminence prediction, raw telemetry achieved an AUROC of 0.689, while the full representation including behavioral-sequence features achieved 0.719. Observable metrics alone performed best at 0.726, an absolute improvement of 0.037 over raw telemetry."

pred-2¶

David Barron, Xiaohang Tang, Rezky Dwisantika, Minsun Kim, David H. Smith IV, Jiaming Cui, and Yan Chen. 2026. TutorTrace: A Dataset and Taxonomy for Classifying Learner Behavioral States during AI-Assisted Programming Education. https://arxiv.org/abs/2608.26184

q2 · i? · associational · r2

Same held-out Random Forest protocol for help-seeking-type prediction (guided vs dependent within 15 seconds), labels generated by GPT-4o with human-rater agreement κ=.897 and κ=.709/.690 between GPT-4o and the raters. Observable metrics reached 0.717.

"Raw telemetry achieved an AUROC of 0.690, while the full representation including behavioral-sequence features achieved 0.705. Observable metrics alone performed best at 0.717, an absolute improvement of 0.027 over raw telemetry."

Discussion¶

Related Claims¶

• First queries from Cold Start learners were overwhelmingly dependent (96.7%), contrasting with mostly guided first queries from Oriented (69.1%) and Struggling (71.4%) learners — related
• Window 1 profiles complicate the assumption that greater pre-query effort signals better progress: Cold Start learners completed at 97% while Struggling learners completed at 67% — related