Question type, stem type, MCQ source, and web accessibility did not independently explain LLM accuracy heterogeneity in biomedical MCQ meta‐regressions¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q4
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q4 i? In meta‐regressions on the full OpenAI dataset, none of question type, stem type, MCQ source, or inferred web accessibility explained more than 3% of heterogeneity (p = 0.09–0.95). → Cheverko CM 2026

Evidence¶

Cheverko CM 2026¶

Cheverko CM, Mavrych V, Bolgova O, Mohamed FRR, Westrick J, Juarez L, Rush E, Solka KA, Doubleday AF, Byram JN, Becker R, Gomez V, Ganeng BKA, Hoffman LA, Roach VA, Brown KM, DeVaul N, Garnett CN, Herriott HL, Lufler RS, Mussell JC, Balta JY, Pascoe MA, Middleton JW, Duffy S, Stephens GC, Wilson AB. (2026). The performance of ChatGPT and other large language models on multiple‐choice questions in biomedical disciplines: A meta‐analysis. Anatomical Sciences Education. https://doi.org/10.1002/ase.70262

q4 · i? · quant-synthesis · r2

Meta‐regressions on the entire OpenAI dataset restricted to studies with clearly defined or quantifiable covariate classifications. The article reports that "None of the additional covariates independently explained more than 3% of the heterogeneity observed in performance accuracy (p = 0.09–0.95)"; no effect size printed.

"None of the additional covariates independently explained more than 3% of the heterogeneity observed in performance accuracy (p = 0.09–0.95)."

Discussion¶

Related Claims¶

• LLM version progression explains about half of the variance in biomedical MCQ accuracy, with GPT‐4o_o1 over six times as likely as GPT‐3/3.5 to answer correctly — related
• GPT‐4 through o1 models show significantly lower pooled accuracy in pathology (75%) than in other biomedical disciplines — related
• None of five moderators (pilot study, retention test, word-selection method, vocabulary outcome, number of words taught) significantly explained heterogeneity in effect sizes — related
• Newer OpenAI LLM versions achieve significantly higher pooled accuracy on biomedical MCQs than earlier versions (GPT‐4o–o1: 90%; GPT‐4: 82%; GPT‐3/3.5: 59%) — related
• Newer Anthropic and DeepSeek models approach GPT‐4o–o1 accuracy on biomedical MCQs, while Google Gemini and Microsoft Copilot trail the leading companies — related
• GPT‐4 through o1 models achieve pooled accuracy of 87% in the anatomical sciences and ≥83% in five of six biomedical disciplines, consistently above older versions — related