Newer OpenAI LLM versions achieve significantly higher pooled accuracy on biomedical MCQs than earlier versions (GPT‐4o–o1: 90%; GPT‐4: 82%; GPT‐3/3.5: 59%)¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q4
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q4 i? GPT‐4o through GPT o1 achieved 90.1% pooled accuracy on text‐based biomedical MCQs, significantly higher than GPT‐4 (81.9%) and GPT‐3/3.5 (59.3%). → Cheverko CM 2026

Evidence¶

Cheverko CM 2026¶

Cheverko CM, Mavrych V, Bolgova O, Mohamed FRR, Westrick J, Juarez L, Rush E, Solka KA, Doubleday AF, Byram JN, Becker R, Gomez V, Ganeng BKA, Hoffman LA, Roach VA, Brown KM, DeVaul N, Garnett CN, Herriott HL, Lufler RS, Mussell JC, Balta JY, Pascoe MA, Middleton JW, Duffy S, Stephens GC, Wilson AB. (2026). The performance of ChatGPT and other large language models on multiple‐choice questions in biomedical disciplines: A meta‐analysis. Anatomical Sciences Education. https://doi.org/10.1002/ase.70262

q4 · i? · quant-synthesis · r2

Random‐effects proportional meta‐analysis of 41 articles comprising 207 proportions of LLM accuracy on text‐based biomedical MCQs. The analysis reports "a 90.1% pooled accuracy (CI: 88.0%–91.9%)" for GPT‐4o_o1 versus GPT‐4 (81.9%) and GPT‐3/3.5 (59.3%), p < 0.001. No standardised effect size printed for this contrast.

"Meta‐analytic findings revealed that GPT‐4o_o1 versions achieved a 90.1% pooled accuracy (CI: 88.0%–91.9%), representing a statistically significant advancement (p < 0.001) from previous versions [GPT‐4 = 81.9% (CI: 78.4%–85.0%); GPT‐3/3.5 = 59.3% (CI: 54.2%–64.3%)]."

Discussion¶

Related Claims¶

• GPT‐4 through o1 models achieve pooled accuracy of 87% in the anatomical sciences and ≥83% in five of six biomedical disciplines, consistently above older versions — related
• Newer Anthropic and DeepSeek models approach GPT‐4o–o1 accuracy on biomedical MCQs, while Google Gemini and Microsoft Copilot trail the leading companies — related
• LLM version progression explains about half of the variance in biomedical MCQ accuracy, with GPT‐4o_o1 over six times as likely as GPT‐3/3.5 to answer correctly — related
• GPT‐4 through o1 models show significantly lower pooled accuracy in pathology (75%) than in other biomedical disciplines — a narrower finding that bears on this claim
• Question type, stem type, MCQ source, and web accessibility did not independently explain LLM accuracy heterogeneity in biomedical MCQ meta‐regressions — related
• GPT-4 achieves passing accuracy on USMLE Steps 1–3 (mean 86%), significantly exceeding GPT-3.5 (mean 56%) — related
• Direct comparisons confirm ChatGPT-4 significantly outperformed earlier versions and some competitor models on dental examination questions — a narrower finding that bears on this claim
• Studies using both text- and image-based questions show higher pooled LLM accuracy (78%) than text-only studies (71%) — related
• Newer LLM versions generally outperformed older versions on dental examination questions — a narrower finding that bears on this claim
• GPT-3.5 achieves an integrated accuracy of 54% on dental licensing examinations — related
• LLMs perform worse on dental licensing examinations than on medical licensing examinations — related