Education algorithm fairness is measured with a diverse set of metrics, most commonly ABROCA, TPR/TNR, FPR/FNR disparity metrics, and group difference in performance¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Across the 12 reviewed studies, common fairness measurement methods were ABROCA, true/false positive rates, disparity metrics, and group difference in performance, with UPAR and counterfactual fairness as other methods. → Jamiu Adekunle Idowu 2024

Evidence¶

Jamiu Adekunle Idowu 2024¶

Jamiu Adekunle Idowu. (2024). Debiasing Education Algorithms. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00389-4

q2 · i? · review · r2

Synthesis of the 12 eligible studies in the SLR, mapping which fairness metric each study used (Table 3). The review concludes "there is no one-size-fits-all approach to fairness assessment" and that metric choice should align with each algorithm's goals and context.

"Clearly, this SLR reveals the diverse nature of metrics for measuring education algorithmic fairness with common methods being ABROCA (S1, S2 & S6), true positive rate and true negative rate (S5 & S9), false positive rate, false negative rate, and disparity metrics (S7 & S10) as well as group difference in performance (S4, S8, S9, S10 & S11)."

Discussion¶

Related Claims¶

• Preprocessing bias mitigation (DIR, RW, SUP) reduced subgroup disparities in TPR on OULAD, with effectiveness varying by demographic attribute — related
• Standard ML models predicting student pass/fail from VLE data exhibit biased true-positive rates across demographic subgroups — related
• Post-processing mitigation with ThresholdOptimizer reduced bias on most fairness metrics while maintaining accuracy reasonably well, with a slight performance drop — related
• Calibrated Equalized Odds Post-processing showed lower balanced accuracy and higher EOD than the preprocessing techniques DIR and RW in student performance prediction — related