Using demographic variables as predictors risks reinforcing biases embedded in training labels, including self-fulfilling prophecies¶

Claim · All claims
Evidence · 1 study (2 entries) · theoretical r3 · q1
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q1 i? When training labels encode biased stakeholder decisions, including demographics in prediction may reinforce the beliefs producing those biased decisions. → Baker 2023 q1 i? There is a risk of a self-fulfilling prophecy where the instructor believes race causes worse outcomes and acts to bring that about. → Baker 2023 (2)

Evidence¶

Baker 2023¶

Baker, R. S., Esbenshade, L., Vitale, J., & Karumbaiah, S. (2023). Using Demographic Data as Predictor Variables: a Questionable Choice. Journal of Educational Data Mining, 15(2). https://jedm.educationaldatamining.org/index.php/JEDM/article/view/619 doi:10.35542/osf.io/y4wvj

q1 · i? · theoretical · r3

Theoretical argument (type e) in section 4.2: "if there is direct bias in the decisions that the model is trying to predict, including these direct biases in prediction may reinforce the beliefs that produce those biased decisions", quoting Paquette et al. (2020) on grade-lifting discretion.

"In other words, if there is direct bias in the decisions that the model is trying to predict, including these direct biases in prediction may reinforce the beliefs that produce those biased decisions."

Baker 2023 (2)¶

Baker, R. S., Esbenshade, L., Vitale, J., & Karumbaiah, S. (2023). Using Demographic Data as Predictor Variables: a Questionable Choice. Journal of Educational Data Mining, 15(2). https://jedm.educationaldatamining.org/index.php/JEDM/article/view/619. doi:10.35542/osf.io/y4wvj

q1 · i? · theoretical · r3

The authors warn, citing Rosenthal & Jacobson (1968), of "a risk of creating a self-fulfilling prophecy" when instructors are told a student will perform worse because of group membership; they also note the Wolff et al. (2013) race variable case as an example of models encoding biased instructor decisions.

"There is also a risk of creating a self-fulfilling prophecy (Rosenthal & Jacobson, 1968), where the instructor believes race causes worse outcomes and therefore acts in ways that bring about that outcome."

Discussion¶

Related Claims¶

• The argument that excluding demographic predictors amounts to color-blind racism is contested: inclusion can also produce racist decision-making — related
• Heterogeneity within demographic categories means group variables as predictors can disadvantage atypical group members and underrepresented groups — related
• Including demographic variables as predictors can reduce actionability by de-emphasizing correlated actionable risk factors and by discounting intervention effects — related
• Evidence is mixed on whether including demographic variables as predictors improves model performance in educational prediction — a narrower finding that bears on this claim
• Incorporating more learning activity data reduced the potential bias caused by overreliance on demographic information — a narrower finding that bears on this claim
• Training constraints during AI development create potential biases that may be amplified by the population data used — related
• Teacher expectation bias predicted Dutch secondary students' performance up to five years later, after controls for prior achievement, IQ and motivation, in one large correlational cohort study — a narrower finding that bears on this claim
• Standard ML models predicting student pass/fail from VLE data exhibit biased true-positive rates across demographic subgroups — related
• Even with skill-based qualifications, LLM team assignment remains gender-biased: when two teams are equally valid, models favor Core Development for male personas over Interface Design — related
• Teacher expectancy effects: seeded expectations led treated students to outperform peers (as reported by the review) — related