Rank-averaged ensembling stabilized predictions but did not outperform the single base model¶

Claim · All claims
Evidence · 1 study · design r2 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q3 i? The ensemble achieved the best public evaluation AUC and second-best private AUC, but did not perform better than Model 1. → Lu 2024

Evidence¶

Lu 2024¶

Lu, Y., Tong, L., & Cheng, Y. (2024). Advanced Knowledge Tracing: Incorporating Process Data and Curricula Information via an Attention-Based Framework for Accuracy and Interpretability. Journal of Educational Data Mining, 16(2). https://osf.io/mdpzc/ doi:10.31219/osf.io/3k7z2

q3 · i? · the article prints no effect size for this finding · design · r2

Results section, Table 4: the rank-averaging ensemble of five trained models reached public AUC 0.79038 and private 0.78969. The authors report "the ensemble model does not perform better than Model 1, suggesting that we can just use Model 1 in practice".

"However, note that the ensemble model does not perform better than Model 1, suggesting that we can just use Model 1 in practice."

Discussion¶

Related Claims¶

• Ablation study: removing any component lowers evaluation AUC, and removing all additional features yields the lowest public and private AUCs — related
• The single-head base model (Model 1) outperformed the more complex Model 2, suggesting overfitting in the larger architecture — related
• The proposed Transformer-based framework achieved first place in the EDM Cup 2023 with an AUC of 78.969% on the private evaluation dataset — related
• Both pedagogy-expertise-weighted and unanimous-vote ensembles frequently worsen LLM alignment with student learning — related
• Ensemble modeling of multiple LLM scorers yields performance gains in automated divergent thinking scoring — reports the opposite
• Majority-vote ensembling across three runs produced only modest accuracy gains (1% to 5%) with no model showing degradation — related
• A stacking ensemble of LG, SVM, NB, KNN and XGB achieved the best fine-tuning F1-score (0.88), outperforming individual models such as SVM (0.86) — related
• A Soft-Voting Ensemble achieves the best behavioral-dataset attrition prediction (CV F1 = 0.737, test recall = 0.769) — related
• Ensembles of operational raters outperform the LLMs: two raters suffice on average across aspects, but larger ensembles are needed for individual aspects — related
• Model self-confidence, weighted probabilistic scoring, and ensemble modeling each significantly improve LLM-based divergent thinking scoring, raising correlation with human judges from r = 0.781 to r = 0.823 — related