Marginal luck probabilities diagnose question calibration, flagging questions 7–9 as prone to over-performance and question 12 as particularly challenging¶

Claim · All claims
Evidence · 1 study · theoretical r3 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? The marginal probability of the luck variable provides a diagnostic of question informativeness: questions 7, 8 and 9 show high probability of very good luck (above 30%), suggesting poor calibration, while question 12 shows bad-luck probability up to 0.76, indicating underperformance is more likely than over-performance. → Francesca Mangili 2026

Evidence¶

Francesca Mangili 2026¶

Francesca Mangili, Alessandro Antonucci, Rafael Cabañas de Paz. (2026). Causal Modelling of Support Interventions for Student Competency Assessment. arXiv. https://arxiv.org/abs/2608.24632 doi:10.48550/arxiv.2608.24632

q2 · i? · theoretical · r3

Group-level marginal inference on the luck variables of the CAT causal model, reported in Table 1, based on the assessment data from 109 students. The first six questions appear more informative than Q7–Q9, since very good luck probability never exceeds 15% in the former but can exceed 30% in the latter.

"it is very unlikely to experience good or very good luck inQ 12, while the probability of bad luck can be as high as 0.76. This suggests that the scheme is particularly challenging, making underperformance more likely than over-performance."

Discussion¶

Related Claims¶

• Group-level marginal inferences identify population propensity and autonomy but are nearly vacuous for higher algorithmic skill levels — related
• Counterfactual probabilities of necessity and sufficiency quantify the causal effect of hints and skills on task performance — related
• Individualised counterfactual inferences estimate whether a specific student would have performed differently under different help conditions — related