AI-generated questions showed slightly higher average discrimination than standardized test questions (αAI = 1.3 vs αSTD = 1.2), with moderate evidence¶
Claim · All claims
Evidence · 1 study · associationalr2·q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? AI-generated questions averaged slightly higher IRT discrimination than standardized test items; the posterior difference favors AI items with Pr(δα > 0) ≈ 0.85, and both types averaged 'moderate' discriminating power. → Calvin Isley 2025
Evidence¶
Calvin Isley 2025¶
Calvin Isley, Joshua Gilbert, Evangelos Kassos, Michaela Kocher, Allen Nie, Emma Brunskill, Ben Domingue, Jake Hofman, Joscha Legewie, Teddy Svoronos, Charlotte Tuminelli, Sharad Goel. (2025). Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study. https://arxiv.org/abs/2508.08314 doi:10.48550/arxiv.2508.08314
q2 · i? · associational · r2
IRT discrimination estimates from the same field-study 2PL model, comparing AI-generated and standardized items; the article reports E[δα] = 0.09 with 95% credible interval [−.0918, .253] and Pr(δα > 0) ≈ 0.85, calling this moderate evidence.
"The AI-generated questions had an average discrimination of ¯αAI = 1.3 across items. In contrast, standardized test questions were slightly less discriminating, with an average of ¯αSTD = 1.2."
Discussion¶
Related Claims¶
- AI-generated exam questions were somewhat easier than standardized-test questions, with average difficulty βAI = −0.45 versus βSTD = 0.35 — related
- AI-generated 10-question exams provided higher maximum test information and reliability (Imax = 3.85, R = 0.79) than the standardized exams (Imax = 2.61, R = 0.72) — related
- For all θ < 0.75, AI-generated exams were on average more informative than the standardized exams, with I(0) = 3.36 versus 2.5 — related
- A larger share of AI-generated questions were highly or very highly discriminating (36%) compared to standardized test questions (21%) — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
ai-items-slightly-more-discriminating |
title |
AI-generated questions showed slightly higher average discrimination than standardized test questions (αAI = 1.3 vs αSTD = 1.2), with moderate evidence |
description |
AI-generated questions showed slightly higher average discrimination than standardized test questions (αAI = 1.3 vs αSTD = 1.2), with moderate evidence |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2025 |
recheck_by |
2028 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — calvin-isley-2025 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.