Including demographic variables as predictors can reduce actionability by de-emphasizing correlated actionable risk factors and by discounting intervention effects¶

Claim · All claims
Evidence · 1 study (3 entries) · theoretical r3 · q1
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q1 i? Including demographic variables can cause models to de-emphasize correlated risk factors more directly aligned with a student's particular needs. → Baker 2023 q1 i? Demographic-variable-based risk predictions risk discounting the impact of interventions because the static demographic factor remains in the model. → Baker 2023 (3)

Evidence¶

Baker 2023¶

Baker, R. S., Esbenshade, L., Vitale, J., & Karumbaiah, S. (2023). Using Demographic Data as Predictor Variables: a Questionable Choice. Journal of Educational Data Mining, 15(2). https://jedm.educationaldatamining.org/index.php/JEDM/article/view/619 doi:10.35542/osf.io/y4wvj

q1 · i? · theoretical · r3

Theoretical argument (type e) in section 4.1: because machine learning algorithms de-emphasize variables without added predictive power, "including demographic variables in models can cause the model to de-emphasize correlated risk factors" aligned with student needs, occluding the appropriate intervention.

"As such, including demographic variables in models can cause the model to de-emphasize correlated risk factors that are more directly aligned with the particular needs of the student – even if the demographic variable achieves only slightly better fit than the correlated variables or achieves worse fit than the correlated variables combined."

Baker 2023 (2)¶

Baker, R. S., Esbenshade, L., Vitale, J., & Karumbaiah, S. (2023). Using Demographic Data as Predictor Variables: a Questionable Choice. Journal of Educational Data Mining, 15(2). https://jedm.educationaldatamining.org/index.php/JEDM/article/view/619. doi:10.35542/osf.io/y4wvj

q1 · i? · theoretical · r3

Supporting real-world observation attributed to Feathers (2022): single demographic variables accounted for "over a quarter of the predictive power of the models" in deployed systems, illustrating the de-emphasis concern.

"Real-world models that incorporate demographic variables often emphasize those variables over other factors, in cases with single demographic variables accounting for over a quarter of the predictive power of the models (see tables in Feathers, 2022)."

Baker 2023 (3)¶

Baker, R. S., Esbenshade, L., Vitale, J., & Karumbaiah, S. (2023). Using Demographic Data as Predictor Variables: a Questionable Choice. Journal of Educational Data Mining, 15(2). https://jedm.educationaldatamining.org/index.php/JEDM/article/view/619. doi:10.35542/osf.io/y4wvj

q1 · i? · theoretical · r3

The authors argue such a model embeds a form of demographic destiny: it "risks discounting the impact of the intervention" since the static demographic factor stays in the model, actively ignoring progress schools make and obscuring students in the group who still need support.

"a risk prediction model based on demographic variables risks discounting the impact of the intervention on the student’s graduation risk as it continues to include and make predictions based on the static racial factor."

Discussion¶

Related Claims¶

• Both prior performance flags and the machine learning algorithm are less accurate when predicting outcomes for students who are Black — the general claim this finding bears on, with the studies behind it
• The argument that excluding demographic predictors amounts to color-blind racism is contested: inclusion can also produce racist decision-making — related
• Heterogeneity within demographic categories means group variables as predictors can disadvantage atypical group members and underrepresented groups — related
• Using demographic variables as predictors risks reinforcing biases embedded in training labels, including self-fulfilling prophecies — related
• In descriptive logit models, demographic variables predict dropout at enrollment (e.g., males have a 60% higher chance of dropping out than females), but most lose significance once first-semester performance data is controlled — related
• Evidence is mixed on whether including demographic variables as predictors improves model performance in educational prediction — a narrower finding that bears on this claim
• Using predictive analytics while ignoring teacher knowledge may misidentify students at risk of dropping out and negatively influence teacher views — related
• Prior research established correlations between observable student experiences such as frustration and clickstream data in ASSISTments, enabling experience-variable equity research without demographic data — related
• Incorporating more learning activity data reduced the potential bias caused by overreliance on demographic information — a narrower finding that bears on this claim
• The model was validated on the complete public school network of the Calabria region using real institutional, territorial, and demographic data — related