Post-hoc fairness interventions on a vendor-controlled EWS redistributed disparities across demographic groups without consistently reducing them¶

Claim · All claims
Evidence · 1 study · causal r? · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across six post-hoc methods applied to calibrated EWS outputs under simulated procurement constraints, fairness interventions redistributed group disparities rather than consistently reducing them. → McConvey 2026

Evidence¶

McConvey 2026¶

McConvey, K., Zhai, A., Li, R., & Guha, S. (2026). Fairness Theatre: Evaluating Post-Hoc Fairness Interventions in Vendor-Controlled Early Warning Systems. arXiv. https://arxiv.org/abs/2609.38552 doi:10.48550/arxiv.2609.38552

q2 · i? · causal · r?

The article's abstract reports the headline outcome of its comparative evaluation of six post-hoc fairness methods on a research EWS replica built from 168,550 student records: "Interventions redistributed disparities without consistently reducing them." The full results section was not available in the supplied text, so this rests on the article's own summary statement rather than its printed tables.

"Interventions redistributed disparities without consistently reducing them. Two implementations favored already-advantaged groups because they used group size to define disadvantage; small, marginalized groups remained poorly served."

Discussion¶

Related Claims¶

• Both prior performance flags and the machine learning algorithm are less accurate when predicting outcomes for students who are Black — the general claim this finding bears on, with the studies behind it
• Two of six evaluated interventions directed corrections toward already-advantaged groups because disadvantage was operationalized using group size — related
• When demographic groups have different base rates, no method can satisfy calibration, equalized odds, and statistical parity simultaneously — a broader claim this one bears on
• Preprocessing bias mitigation (DIR, RW, SUP) reduced subgroup disparities in TPR on OULAD, with effectiveness varying by demographic attribute — related
• Calibrated Equalized Odds Post-processing showed lower balanced accuracy and higher EOD than the preprocessing techniques DIR and RW in student performance prediction — a narrower finding that bears on this claim
• Post-processing mitigation with ThresholdOptimizer reduced bias on most fairness metrics while maintaining accuracy reasonably well, with a slight performance drop — a narrower finding that bears on this claim
• The xOrder fairness method proved inapplicable at institutional scale, with runtime scaling of approximately n^2.36 — related
• Standard ML models predicting student pass/fail from VLE data exhibit biased true-positive rates across demographic subgroups — related
• Balancing the dataset significantly improved model predictive performance but increased bias under demographic parity — related
• Fairness of AI outputs remains under-evaluated in edtech research — related
• Bias in predicting learning productivity, measured as accuracy differences across gender and parental educational background, was highly dependent on the ML method used — related