Independent reviewers could not reliably identify whether SBMCQs were faculty- or ChatGPT-generated¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person
Design limits · Chauhan 2025: no comparison group (read from the study by two models, 2026-10-27)

Subclaims¶

q2 i? None of the five blinded external reviewers achieved a correct guess rate of 90% or above when judging item source. → Chauhan 2025

Evidence¶

Chauhan 2025¶

Chauhan, A., Khaliq, F., & Nayak, K. R. (2025). Assessing Quality of Scenario-Based Multiple-Choice Questions in Physiology: Faculty-Generated vs. ChatGPT-Generated Questions among Phase I Medical Students. International Journal of Artificial Intelligence in Education, 35, 2315–2344. https://doi.org/10.1007/s40593-025-00471-z

q2 · i? · causal · r2

Blind-source judgment task during the development phase, in which five independent external reviewers rated 120 coded, randomized SBMCQs and guessed each item's source. The article reports no reviewer reached a 90% correct guess rate; no effect size is printed.

"the study's results indicated that there was no correlation between the guesses made by the independent faculty reviewers on the source of SBMCQ by faculty or ChatGPT. None of the five reviewers achieved a correct guess rate of 90% or above."

Discussion¶

Related Claims¶

• Faculty focus groups raised concerns about authenticity, validity, and reliability of ChatGPT-generated SBMCQs, including cultural and linguistic bias — related