AI systems relying on black-box models should not be used for high-stakes applications¶

Claim · All claims
Evidence · 1 study · theoretical r2 · q1
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q1 i? The report asserts that AI systems relying on black-box models, whose hidden-layer computations are intractable to human interpretation, should not be used for high-stakes applications. → Mills 2024

Evidence¶

Mills 2024¶

Mills, K., Ruiz, P., Lee, K., Coenraad, M., Fusco, J., Roschelle, J., & Weisgrau, J. (2024). AI Literacy: A Framework to Understand, Evaluate, and Use Emerging Technology. Digital Promise. https://doi.org/10.51388/20.500.12265/218

q1 · i? · theoretical · r2

The report's mechanistic explanation of artificial neural networks—arranged in input, hidden, and output layers with weighted associations—states that hidden layers perform statistical functions that are intractable, making the model a 'black box.'

"Given AI systems and tools rely on black boxes that do not allow humans to understand how outputs are decided on, these AI systems and tools should not be used for high-stakes applications."

Discussion¶

Related Claims¶

• LLM-assisted inductive qualitative coding carries risks of superficial themes, broad or redundant codes, and hallucinated interpretations, so LLMs should augment rather than replace human researchers — related
• Decision trees achieve competitive accuracy with black-box models (LR, RF, SVM), which performed slightly better but lack interpretability — related
• Neural networks on linguistic features classify Key Stages poorly, with deeper networks performing worse; the best (one hidden layer, 175 ReLU) reached F1 = 0.392 — related
• Algorithmically inferred measurement functions may intensify, rather than resolve, the problem of nomic measurement in analyzing student learning — related
• Longitudinal studies of trust and reliance in AI advice examine whether users rely appropriately over time but not whether independent reasoning capacity is maintained — related