For all θ < 0.75, AI-generated exams were on average more informative than the standardized exams, with I(0) = 3.36 versus 2.5¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Across the ability spectrum below θ = 0.75, AI-generated exams provided greater average information than standardized exams, including at θ = 0 (I(0) = 3.36 vs 2.5). → Calvin Isley 2025

Evidence¶

Calvin Isley 2025¶

Calvin Isley, Joshua Gilbert, Evangelos Kassos, Michaela Kocher, Allen Nie, Emma Brunskill, Ben Domingue, Jake Hofman, Joscha Legewie, Teddy Svoronos, Charlotte Tuminelli, Sharad Goel. (2025). Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study. https://arxiv.org/abs/2508.08314 doi:10.48550/arxiv.2508.08314

q2 · i? · associational · r2

Test-information comparison at specific ability levels from the field-study IRT analysis; the article reports that "for all θ < .75" the AI-generated exams were on average more informative than the standardized-exam tests for students in the study population.

"In particular, at θ = 0 , the AI exams have an aver- age I(0) = 3 .36, whereas the standardized test questions have an averageI(0) = 2.5."

Discussion¶

Related Claims¶

• AI-generated exam questions were somewhat easier than standardized-test questions, with average difficulty βAI = −0.45 versus βSTD = 0.35 — related
• AI-generated 10-question exams provided higher maximum test information and reliability (Imax = 3.85, R = 0.79) than the standardized exams (Imax = 2.61, R = 0.72) — related
• A larger share of AI-generated questions were highly or very highly discriminating (36%) compared to standardized test questions (21%) — related
• AI-generated questions showed slightly higher average discrimination than standardized test questions (αAI = 1.3 vs αSTD = 1.2), with moderate evidence — related