CatBoost achieved the best classification performance among tested algorithms for predicting course completion risk (F-measure .77, accuracy 78%, AUC .87)¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person · recheck by 2026: tool and model accuracy changes quickly

Subclaims¶

q2 i? CatBoost outperformed k-nearest neighbours, Naïve Bayes, logistic regression, and random forest on F-measure, accuracy, and AUC in a modified 10-fold cross-validation across seven courses. → Gomathy Ramaswami 2023

Evidence¶

Gomathy Ramaswami 2023¶

Gomathy Ramaswami, Teo Susnjak and Anuradha Mathrani. (2023). Effectiveness of a Learning Analytics Dashboard for Increasing Student Engagement Levels. Journal of Learning Analytics, 10(3). https://doi.org/10.18608/jla.2023.7935

q2 · i? · causal · r1

Model evaluation using a modified k-fold cross-validation training on nine course deliveries and testing on the hold-out course, repeated 10 times. CatBoost scored F-measure .77±.02, accuracy 78±2.1%, AUC .87±.02, the highest among the five algorithms compared in Table 2.

"Given that CatBoost achieved the highest scores, the models from this algorithm were selected to identify the at-risk students and to display its outputs on the dashboard."

Discussion¶

Related Claims¶

• DynEmb outperforms BMF and DKT baselines in future response prediction across five tutoring datasets, with AUC improvement up to 5.43% in the New User setting — related
• A random forest classified flagged versus non-flagged word problems with the best accuracy of five tested models (AUC = 0.75) — related
• Random Forest is the most commonly used NLP-supporting algorithm in LA, while deep learning models (LSTM, BERT) have been widely adopted since 2021 — related
• Baseline LR and XGBoost models achieve comparable predictive performance on OULAD, with XGBoost slightly higher AUC-ROC — related
• Gradient boosting and extreme gradient boosting are the highest-performing classifiers for predicting enrollment, with balanced accuracy 0.91, F-score 0.88 and AUC 0.96 — related
• Random forest algorithms yielded the best classification performance in K–8 MMLA studies comparing multiple machine learning models — reports the opposite
• Simple classifiers (Logistic Regression, Naive Bayes) outperformed tree-based models on this small dataset — reports the opposite
• Question and observed-coverage features retrospectively predict high-divergence questions with AUC 0.774, but the model is not validated for new-query routing — related
• k-nearest neighbours achieves the highest recall of the tested models, making it better suited when identifying which students know a topic matters — related
• Machine learning methods outperformed logistic regression in classification accuracy across reviewed studies — a broader claim this one bears on
• Decision trees achieve competitive accuracy with black-box models (LR, RF, SVM), which performed slightly better but lack interpretability — related
• Boosting models (XGBoost) achieved the highest predictive accuracy among the compared ML models, while logistic regression showed the lowest — related