Expert-panel human-review workflow for AI-generated MCQ drafting in a high-stakes exam

Design · All designs
Evidence · 2 claims (2 for) · 1 study (1 associational), q3 · 0 of 1 report an effect size · 2 claims rest on one study

Description

A structured workflow in which LLMs (ChatGPT-4o and Gemini 1.5 Pro) received the full blended-learning curriculum and generated MCQs via sequential prompting, followed by expert-panel review. The panel "accepted 82% of the questions", applied "limited expert refinement" to retained items, and excluded items needing substantial revision; 30 AI questions (18 ChatGPT, 12 Gemini) entered the exam. The authors position it so "AI-generated MCQs serve as template, while humans remain responsible for quality control".

Design Implications

Context

Requirements

• Human expert panel review of all generated items before exam use
• Provision of the full curriculum and learning materials to the LLMs for generation

Constraints

• 18.2% of initially generated MCQs had to be eliminated, showing the necessity of human oversight
• A visible bias underestimating FM competency was observed during human evaluation, e.g. distractors suggesting referrals for abdominal ultrasound, a common FM task

Target Learners

• fifth-year undergraduate medical students

Learning Goals

• Family Medicine clinical decision-making and clinical reasoning assessment

Claims

• Recognition Does Not Differ Ai Vs Nle [+M]
• Perceived Alignment Does Not Differ Ai Vs Nle [+M]

Related Designs

Examples

-

Key Sources

• Vogt P, Wolf N, Jordan S, Volz-Willems S, Jäger J, Dupont F. (2026). Can students identify AI? – A cross-sectional quantitative study about AI recognition in tablet-based MCQ assessment among fifth-year undergraduate medical students at Saarland University, Germany. BMC Medical Education. https://doi.org/10.1186/s12909-026-10410-8

Page metadata

Field	Value
type	design
id	expert-panel-ai-mcq-review-workflow
title	Expert-panel human-review workflow for AI-generated MCQ drafting in a high-stakes exam
description	A structured workflow in which LLMs (ChatGPT-4o and Gemini 1.5 Pro) received the full blended-learning curriculum and generated MCQs via sequential prompting, followed by expert-panel review.
status	review
generated	by process:settle-candidates on 2026-10-12
trust	unverified — no one has confirmed this page's content
sources	1 — vogt-p-2026

Frontmatter is the page's machine-readable record — see the Schema & Guide.