Algorithmically inferred measurement functions may intensify, rather than resolve, the problem of nomic measurement in analyzing student learning¶

Claim · All claims
Evidence · 1 study · theoretical r2 · q1
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q1 i? Because the mapping between observable student data and cognitive constructs emerges from training data and optimization rather than explicit researcher positing, its validity is often obscured by black-box algorithms. → Paul P. Martin 2026

Evidence¶

Paul P. Martin 2026¶

Paul P. Martin, Marvin Rost, Jenna Koenen, Nicole Graulich. (2026). Amid an Epistemic Iteration: How AI Methodologies May Transform the Nature of Science Education Research. Science & Education. https://doi.org/10.1007/s11191-026-00789-7

q1 · i? · theoretical · r2

Conceptual analysis in Phase 2 (instrumentation and measurement), applying Chang's problem of nomic measurement to supervised and unsupervised learning, where the measurement function "emerges from the training data and the optimization procedures of the chosen algorithm." This is an analytical argument, not an empirical test.

"AI methodologies may intensify the problem of nomic measurement rather than resolve it, because the validity of the inferred measurement functions is often obscured by a black‑box algorithm that offers no epistemic justification for why student data are mapped onto specific cognitive attributes"

Discussion¶

Related Claims¶

• Implementing AI methodologies introduces a new layer of epistemic dependence and diffused accountability in science education research — related
• BKT-BF suffers high computational cost and does not resolve BKT's identifiability problem, while EM is cheaper but suffers local minima — related
• All three trained classifiers outperformed the zero-rule baseline (28.4% accuracy) for classifying cognitive engagement in discussion posts — related
• GenAI undermines evidence rules by making work products ambiguous between student competence and AI capability, with substitution capacity depending on task cognitive demands — related
• Group-level marginal inferences identify population propensity and autonomy but are nearly vacuous for higher algorithmic skill levels — related
• Algorithmic empathy is culturally fragile: emotional inference errors in socio-emotional AI can invalidate teachers' experiences, so systems require local validation and participatory design — related
• Formalisation of theory offers generative models, counterfactual reasoning, and optimal-performance yardsticks unavailable to verbal accounts — related
• AI systems relying on black-box models should not be used for high-stakes applications — related
• Cognitive process inference in computerized testing rarely rests on strong process data — related