A larger share of AI-generated questions were highly or very highly discriminating (36%) compared to standardized test questions (21%)¶
Claim · All claims
Evidence · 1 study · associationalr2·q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Following the Baker (2001) taxonomy, 36% of AI-generated questions were highly or very highly discriminating versus 21% of standardized test questions, while most items in both sets were 'moderate'. → Calvin Isley 2025
Evidence¶
Calvin Isley 2025¶
Calvin Isley, Joshua Gilbert, Evangelos Kassos, Michaela Kocher, Allen Nie, Emma Brunskill, Ben Domingue, Jake Hofman, Joscha Legewie, Teddy Svoronos, Charlotte Tuminelli, Sharad Goel. (2025). Assessing the Quality of AI-Generated Exams: A Large-Scale Field Study. https://arxiv.org/abs/2508.08314 doi:10.48550/arxiv.2508.08314
q2 · i? · associational · r2
Proportions of items at each Baker (2001) discrimination level, reported in Table 1 from the field-study IRT estimates; the article states AI-generated questions were also "more tightly distributed than the standardized test questions".
"While a large share of both the AI-generated (64%) and standardized test questions (71%) are considered "moderate", 36% of the AI-generated questions are "highly" or "very highly" discriminating compared to 21% of the standardized test questions."
Discussion¶
Related Claims¶
- AI-generated exam questions were somewhat easier than standardized-test questions, with average difficulty βAI = −0.45 versus βSTD = 0.35 — related
- For all θ < 0.75, AI-generated exams were on average more informative than the standardized exams, with I(0) = 3.36 versus 2.5 — related
- AI-generated questions showed slightly higher average discrimination than standardized test questions (αAI = 1.3 vs αSTD = 1.2), with moderate evidence — related
- Benchmark items showed high mean difficulty and low item discrimination, with many trivial items and only one broken item — reports the opposite
- AI-generated 10-question exams provided higher maximum test information and reliability (Imax = 3.85, R = 0.79) than the standardized exams (Imax = 2.61, R = 0.72) — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
ai-items-higher-share-high-discrimination |
title |
A larger share of AI-generated questions were highly or very highly discriminating (36%) compared to standardized test questions (21%) |
description |
A larger share of AI-generated questions were highly or very highly discriminating (36%) compared to standardized test questions (21%) |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2025 |
recheck_by |
2028 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — calvin-isley-2025 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.