Code-DKT's attention softmax applied across the temporal dimension (dim=1) is causally invalid and inflates reported performance; rectifying it to the path dimension (dim=2) removes the artificial gain¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? The official implementation applies Softmax across the time-step dimension, allowing normalization over the entire interaction sequence and introducing temporal data leakage. → Jaewook Kim 2026 q2 i? The authors replicated the original reported gain of approximately 0.03–0.04 AUC over DKT before identifying the flaw. → Jaewook Kim 2026 (2)

Evidence¶

Jaewook Kim 2026¶

Jaewook Kim, Hyeoncheol Kim. (2026). Ensuring Reliability in Programming Knowledge Tracing: A Re-evaluation of Attention-augmented Models and Experimental Protocols. Preprint, International Conference on Intelligent Tutoring Systems (ITS 2026), Springer LNCS. https://arxiv.org/abs/2605.04727 doi:10.48550/arxiv.2605.04727

q2 · i? · causal · r2

Code audit of the official Code-DKT source. The authors argue this "look-ahead" mechanism lets the importance of code features at time t be influenced by future states, and that normalization must instead occur across the path dimension (dim=2).

"the Softmax operation in the attention layer is applied across the temporal dimension, specificallydim=1"

Jaewook Kim 2026 (2)¶

Jaewook Kim, Hyeoncheol Kim. (2026). Ensuring Reliability in Programming Knowledge Tracing: A Re-evaluation of Attention-augmented Models and Experimental Protocols. Preprint, International Conference on Intelligent Tutoring Systems (ITS 2026), Springer LNCS. https://arxiv.org/abs/2605.04727

q2 · i? · causal · r2

Replication of Code-DKT results from the official implementation on CodeWorkout, confirming the reported gain over the DKT baseline before the audit. The authors attribute the gain to "this causal violation" rather than architectural superiority.

"we executed the official source code [16] provided by the authors and successfully replicated the empirical results reported in the original paper, including the performance gain of approximately 0.03–0.04 in AUC"

Discussion¶

Related Claims¶

• Failing to sort student attempt sequences by ServerTimestamp violates temporal causality and can systematically overestimate PKT model performance — related
• Under causally valid, tuned settings, code-level embeddings do not consistently improve performance: rectified Code-DKT underperforms tuned DKT on several assignments, while ECKT remains competitive — related
• Extending maximum sequence length from 50 to 100 does not lead to consistent knowledge-tracing performance improvements; negative changes appear more frequently across models and datasets — related
• Task-specific hyperparameter optimization substantially raises baseline DKT performance, indicating prior DKT-vs-PKT gaps partly reflect insufficient baseline tuning — related