All reviewed studies presented faulty LLM-generated questions deemed inappropriate for medical exams¶
Claim · All claims
Evidence · 1 study · reviewr2·q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Every included study reported some generated questions that were inappropriate for medical exams, and some required additional modification to qualify. → Artsi Yaara 2024
Evidence¶
Artsi Yaara 2024¶
Artsi Yaara, Sorin Vera, Konen Eli, Glicksberg Benjamin S., Nadkarni Girish, Klang Eyal. (2024). Large language models for generating medical examinations: systematic review. BMC Medical Education. https://doi.org/10.1186/s12909-024-05239-y
q2 · i? · review · r2
Cross-study finding from the review's descriptive summary of results, tabulated in its Table 3 of faulty questions (medically irrelevant, invalid, inaccurate, or low-difficulty items). The review states "All studies presented some faulty questions".
"All studies presented some faulty questions that were deemed inappropriate for medical exams. Some questions required additional modifications in order to qualify"
Discussion¶
Related Claims¶
- Blind expert assessment of GPT-4-generated MCQs found the majority valid as exam questions, with problematic-question rates varying by field — reports the opposite
- In the one human-vs-LLM comparison, human-written MCQs outperformed ChatGPT-generated MCQs in quality and validity, while AI was far faster — related
- Most reviewed studies (5 of 8) demonstrated that LLMs can produce MCQs valid for medical exams — reports the opposite
Page metadata
| Field | Value |
|---|---|
status |
review |
generated |
by process:wiki-ingest on 2026-10-31 |
trust |
unverified — no one has confirmed this page's content |
Frontmatter is the page's machine-readable record — see the Schema & Guide.