AI-generated exam questions were somewhat easier than standardized-test questions, with average difficulty βAI = −0.45 versus βSTD = 0.35¶
Claim · All claims
Evidence · 1 study · associationalr2·q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? In the field study, AI-generated questions had lower average IRT difficulty than standardized test questions, and students answered about 60% of AI items correctly versus 39% of standardized items. → Calvin Isley 2025
Evidence¶
Calvin Isley 2025¶
Calvin Isley, Joshua Gilbert, Evangelos Kassos, Michaela Kocher, Allen Nie, Emma Brunskill, Ben Domingue, Jake Hofman, Joscha Legewie, Teddy Svoronos, Charlotte Tuminelli, Sharad Goel. (2025). Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study. https://arxiv.org/abs/2508.08314 doi:10.48550/arxiv.2508.08314
q2 · i? · associational · r2
IRT analysis of field-student responses (2PL hierarchical Bayesian model) estimating item difficulty by exam type; the article reports students correctly answered approximately 60% of AI-generated questions, versus 39% for standardized items, "as shown in Figure 2".
"The average difficulty of the AI-generated questions is ¯βAI =−0.45, indicating that students with estimated abilities somewhat below the average had approximately even odds of answering these questions correctly."
Discussion¶
Related Claims¶
- AI-generated questions showed slightly higher average discrimination than standardized test questions (αAI = 1.3 vs αSTD = 1.2), with moderate evidence — related
- For all θ < 0.75, AI-generated exams were on average more informative than the standardized exams, with I(0) = 3.36 versus 2.5 — related
- A larger share of AI-generated questions were highly or very highly discriminating (36%) compared to standardized test questions (21%) — related
- AI-generated 10-question exams provided higher maximum test information and reliability (Imax = 3.85, R = 0.79) than the standardized exams (Imax = 2.61, R = 0.72) — related
- Exposed item-bank problems did not show an overall 'seen before, hence easier' pattern in this sample — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
ai-exam-items-easier-than-standardized |
title |
AI-generated exam questions were somewhat easier than standardized-test questions, with average difficulty βAI = −0.45 versus βSTD = 0.35 |
description |
AI-generated exam questions were somewhat easier than standardized-test questions, with average difficulty βAI = −0.45 versus βSTD = 0.35 |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2025 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — calvin-isley-2025 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.