Faculty focus groups raised concerns about authenticity, validity, and reliability of ChatGPT-generated SBMCQs, including cultural and linguistic bias¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person
Design limits · Chauhan 2025 (2): no comparison group (read from the study by two models, 2026-10-27)

Subclaims¶

q2 i? FGD participants reported concerns about generic, non-region-specific questions, validity without oversight, and inconsistent difficulty and errors in ChatGPT-generated items. → Chauhan 2025 q2 i? Participants judged that ChatGPT questions fell short in capturing regional and sociocultural context even when the prompt requested it. → Chauhan 2025 (2)

Evidence¶

Chauhan 2025¶

Chauhan, A., Khaliq, F., & Nayak, K. R. (2025). Assessing Quality of Scenario-Based Multiple-Choice Questions in Physiology: Faculty-Generated vs. ChatGPT-Generated Questions among Phase I Medical Students. International Journal of Artificial Intelligence in Education, 35, 2315–2344. https://doi.org/10.1007/s40593-025-00471-z

q2 · i? · qualitative · r2

Thematic analysis of an in-person focus group discussion with five physiology faculty (mean duration 37.15 min), yielding two themes and four subthemes on ChatGPT SBMCQ quality. The reliability-challenges subtheme is quoted; qualitative work, no effect size.

"Participants highlighted concerns regarding variations in question difficulty and identified errors or inaccuracies in ChatGPT-generated content, which could impact the credibility of assessment results."

Chauhan 2025 (2)¶

Chauhan, A., Khaliq, F., & Nayak, K. R. (2025). Assessing Quality of Scenario-Based Multiple-Choice Questions in Physiology: Faculty-Generated vs. ChatGPT-Generated Questions among Phase I Medical Students. International Journal of Artificial Intelligence in Education, 35, 2315–2344. https://doi.org/10.1007/s40593-025-00471-z

q2 · i? · causal · r2

Cultural-bias subtheme from the same FGD thematic analysis: participants judged ChatGPT items as disconnected from local realities despite region-specific prompting, which the authors attribute to predominantly Western training data. Qualitative finding, no effect size.

"the ChatGPT-generated questions often fell short of authentically capturing these elements. Despite attempts to include regional context in the prompt used, the questions produced by ChatGPT appeared disconnected from local realities."

Discussion¶

Related Claims¶

• The most frequently cited ethical concerns about GenAI in research were hallucinated facts, fabricated references, and misleading content — a broader claim this one bears on
• Independent reviewers could not reliably identify whether SBMCQs were faculty- or ChatGPT-generated — related
• The most reported risks of AI tools in science and chemistry education are ethical issues, including gender and racial bias, hallucinations, copyright infringement, and plagiarism — a broader claim this one bears on
• Faculty-generated SBMCQs showed superior discrimination indices and distractor effectiveness compared with ChatGPT-generated SBMCQs — related
• ChatGPT-generated SBMCQs skewed easier than faculty-generated SBMCQs, whose difficulty was predominantly in the appropriate range — related
• ChatGPT-generated SBMCQs contained a higher rate of nonfunctional distractors than faculty-generated SBMCQs (33.75% vs. 13.75%) — related
• Educator- and institutional-level challenges include curricular misalignment, need for teacher mediation, unequal access, privacy concerns, academic integrity risks, and algorithmic bias — a broader claim this one bears on
• Faculty-generated SBMCQs met all three ideal item-analysis criteria more often than ChatGPT-generated SBMCQs (19 vs. 9 of 40) — related