Ablation study: removing any component lowers evaluation AUC, and removing all additional features yields the lowest public and private AUCs¶

Claim · All claims
Evidence · 1 study · design r2 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q3 i? The original Model 1 had the largest best validation, public, and private evaluation AUCs, and removing all additional features produced the lowest public and private AUCs. → Lu 2024

Evidence¶

Lu 2024¶

Lu, Y., Tong, L., & Cheng, Y. (2024). Advanced Knowledge Tracing: Incorporating Process Data and Curricula Information via an Attention-Based Framework for Accuracy and Interpretability. Journal of Educational Data Mining, 16(2). https://osf.io/mdpzc/ doi:10.31219/osf.io/3k7z2

q3 · i? · the article prints no effect size for this finding · design · r2

Ablation study on Model 1 (Table 5), omitting sequence-level embeddings, student/class embeddings, action features, and problem features individually and jointly. Removing all features gave private AUC 0.7802 versus 0.7908 for the original model.

"Table 5 also shows that removing all the additional features in the table will result in both the lowest public and private evaluation AUCs, suggesting the importance of incorporating all the additional features to maximize the performance of the model."

Discussion¶

Related Claims¶

• The single-head base model (Model 1) outperformed the more complex Model 2, suggesting overfitting in the larger architecture — related
• Rank-averaged ensembling stabilized predictions but did not outperform the single base model — related
• The propdec adaptive student feature is highly colinear with a student intercept, capturing student individual differences without student parameters — related
• Ablation of feature-vector models: time-window features add no predictive power to logistic regression but boost a feedforward network, and total count features substantially boost performance on all datasets — related
• Removing the cognitive load module causes the largest ablation performance drop, while state fusion has a smaller effect — related
• Ablation study shows metadata-enriched embeddings produce the largest accuracy drop, while attention masking, KV caching and dynamic batching primarily support inference efficiency — related
• The proposed Transformer-based framework achieved first place in the EDM Cup 2023 with an AUC of 78.969% on the private evaluation dataset — related