Balancing the dataset significantly improved model predictive performance but increased bias under demographic parity¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2 · i3 large
Currency · newest study 2025 · checked against its sources 2026-10-15 (automated, 2 of 2 entries) · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q2 i3 Models trained on the balanced dataset showed significant performance improvements with large effect sizes (larger than 0.80) on each metric at each stage. → Alison Cheng 2025 q2 i3 Training on the balanced dataset produced a significant increase in bias under Demog_Parity_Diffs (p-value < 0.005) at all stages, showing balancing alone does not mitigate bias. → Alison Cheng 2025 (2)

Evidence¶

Alison Cheng 2025¶

Alison Cheng, Bo Pei and Cheng Liu. (2025). Balancing Act: Early, Fair, and Accurate Identification of At-Risk Students. Journal of Learning Analytics, 12(3), 47–65. https://doi.org/10.18608/jla.2025.8761

q2 · i3 · design · r2

Comparison of models trained on RandomOverSampler-balanced versus imbalanced datasets across three stages and four metrics. The article reports "a large Cohen’s d for each metric at each stage (i.e., larger than 0.80)", i.e., Cohen's d larger than 0.80.

"Apart from the significant differences in performance under most metrics, we also found a large Cohen’s d for each metric at each stage (i.e., larger than 0.80), indicating strong effect sizes."

Alison Cheng 2025 (2)¶

Alison Cheng, Bo Pei and Cheng Liu. (2025). Balancing Act: Early, Fair, and Accurate Identification of At-Risk Students. Journal of Learning Analytics, 12(3), 47–65. https://doi.org/10.18608/jla.2025.8761

q2 · i3 · design · r2

Fairness comparison between models trained on balanced and imbalanced datasets showed "a significant increase in bias under the Demog_Parity_Diffs (p-value < 0.005) metric for all stages" (Cohen's d = 5.208, 4.367 and 9.167 at the Demog Stage, Stage 1 and Stage 2). No other metric changed significantly: most showed non-significant increases in bias (with medium to large d on Avg_Odds_Diffs at the Demog Stage, 1.040, and Stage 1, 1.086, and on PPDs at Stage 2, 0.771), except Avg_Odds_Diffs at Stage 2, where "a slight decrease in bias was found (t-statistics = –0.043, Cohen's d = 0.030)".

"Table 3 shows a significant increase in bias under the Demog_Parity_Diffs (p-value < 0.005) metric for all stages."

Discussion¶

Related Claims¶

• Both prior performance flags and the machine learning algorithm are less accurate when predicting outcomes for students who are Black — the general claim this finding bears on, with the studies behind it
• Post-processing mitigation with ThresholdOptimizer reduced bias on most fairness metrics while maintaining accuracy reasonably well, with a slight performance drop — related
• Preprocessing bias mitigation (DIR, RW, SUP) reduced subgroup disparities in TPR on OULAD, with effectiveness varying by demographic attribute — related
• Stage 1 (after the first unit review assignment) is the optimal point for identifying at-risk students, balancing timeliness, accuracy, and fairness — related
• Adding more learning activity data beyond Stage 1 did not significantly improve predictive performance of at-risk identification models — related
• Incorporating more learning activity data reduced the potential bias caused by overreliance on demographic information — related
• Removing the sensitive race feature caused little impact on predictive performance at Stage 1 and no significant fairness differences, with mixed fairness effects — related
• Post-hoc fairness interventions on a vendor-controlled EWS redistributed disparities across demographic groups without consistently reducing them — related
• AI-powered predictive analytics improves academic performance by enabling early identification of at-risk students and timely intervention — related