Exposed item-bank problems did not show an overall 'seen before, hence easier' pattern in this sample¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The 12 item-bank-exposed problems had slightly higher mean raw AI difficulty (2.667 vs 2.582), lower student mean pass rate (36.6% vs 42.8%), and higher mean non-attempt rate (49.6% vs 35.4%) than non-exposed problems. → Hongfei Yan 2026

Evidence¶

Hongfei Yan 2026¶

Hongfei Yan, Jiangkai Xiong, Yiqing Li, and Chong Chen. (2026). From Evaluated Models to Evaluation Aids: A Multi-Evidence Study of LLM-Based Difficulty Calibration for Programming Examinations. https://arxiv.org/abs/2608.07523 doi:10.48550/arxiv.2608.07523

q2 · i? · associational · r2

Descriptive comparison of exposed versus non-exposure-flagged problems in the 79-problem sample; no effect size or significance test is printed, so impact is null. The authors relate this to exposed problems of Class 8 being concentrated in a single class, inseparable from the class effect.

"The mean raw AI difficulty of the 12 item-bank original problems is 2.667, slightly higher than the 2.582 of the 67 non-exposure-flagged problems; their student mean pass rate is 36.6%, lower than the 42.8% of the non-exposure-flagged problems, and their mean non-attempt rate is 49.6%, higher than the 35.4% of the non-exposure-flagged problems."

Discussion¶

Related Claims¶

• Review-based AI overall difficulty correlates strongly with problem-level student pass rate across 79 parallel-class exam problems — related
• AI-generated exam questions were somewhat easier than standardized-test questions, with average difficulty βAI = −0.45 versus βSTD = 0.35 — related