Exploratory analysis found item difficulty differed across sources, driven by the Google Gemini versus NLE contrast¶
Claim · All claims
Evidence · 1 study · associationalr2·q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? Item difficulty differed significantly across ChatGPT, Google Gemini and NLE items, with a pairwise difference between Google Gemini and NLE items. → Vogt P 2026
Evidence¶
Vogt P 2026¶
Vogt P, Wolf N, Jordan S, Volz-Willems S, Jäger J, Dupont F. (2026). Can students identify AI? – A cross-sectional quantitative study about AI recognition in tablet-based MCQ assessment among fifth-year undergraduate medical students at Saarland University, Germany. BMC Medical Education. https://doi.org/10.1186/s12909-026-10410-8
q3 · i? · associational · r2
Post-hoc exploratory Kruskal–Wallis analysis of item difficulty across ChatGPT (Mdn = 0.890), Google Gemini (Mdn = 0.730) and NLE (Mdn = 0.930) items. The omnibus test gave "ε2 = 0.177"; only the Gemini–NLE pairwise contrast was significant. No standardised effect size in d/r terms is printed.
"Item difficulty differed significantly across the three sources (χ 2(2) = 6.71, p = 0.035, ε2 = 0.177). Pairwise comparisons indicated a significant difference between Google Gemini and NLE (p = 0.028), whereas no significant difference was observed between ChatGPT and NLE (p = 0.984)"
Discussion¶
Related Claims¶
- AI-generated assessment items frequently need faculty correction, supporting AI as a draft generator within quality assurance — the general claim this finding bears on, with the studies behind it
- Recognition was not associated with item difficulty — related
- LLM difficulty predictions for fraction items are less stable across repeated runs than for the overall item set — related
- Models with similar rank correlations differ substantially in estimating difficulty magnitude (RMSE and MAE) — related
- Student-perceived curricular alignment did not differ between AI and NLE MCQs or among ChatGPT, Gemini and NLE — related
- Students' recognition did not differ between AI-generated and NLE MCQs — related
- Perceived overall utility of generative AI differs significantly across the five platforms used, favoring platforms with structured task breakdowns — related
- Distractor distribution did not differ across ChatGPT, Google Gemini and NLE MCQs — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
item-difficulty-differs-across-sources-gemini-nle |
title |
Exploratory analysis found item difficulty differed across sources, driven by the Google Gemini versus NLE contrast |
description |
Exploratory analysis found item difficulty differed across sources, driven by the Google Gemini versus NLE contrast |
status |
review |
evidence_strength |
weak |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — vogt-p-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.