GPT‐4 through o1 models show significantly lower pooled accuracy in pathology (75%) than in other biomedical disciplines¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q3 i? GPT‐4_o1 versions achieved 75% pooled accuracy in pathology, significantly lower than other disciplines (meta‐regression p = 0.04). → Cheverko CM 2026

Evidence¶

Cheverko CM 2026¶

Cheverko CM, Mavrych V, Bolgova O, Mohamed FRR, Westrick J, Juarez L, Rush E, Solka KA, Doubleday AF, Byram JN, Becker R, Gomez V, Ganeng BKA, Hoffman LA, Roach VA, Brown KM, DeVaul N, Garnett CN, Herriott HL, Lufler RS, Mussell JC, Balta JY, Pascoe MA, Middleton JW, Duffy S, Stephens GC, Wilson AB. (2026). The performance of ChatGPT and other large language models on multiple‐choice questions in biomedical disciplines: A meta‐analysis. Anatomical Sciences Education. https://doi.org/10.1002/ase.70262

q3 · i? · quant-synthesis · r2

Discipline‐level pooled analysis of GPT‐4_o1 accuracy with meta‐regression comparing pathology against other disciplines; "this difference between pathology and the other disciplines was statistically significant (p = 0.04)". No effect size printed.

"GPT‐4_o1 versions achieved 75% accuracy in pathology. Using meta‐regression, this difference between pathology and the other disciplines was statistically significant (p = 0.04)."

Discussion¶

Related Claims¶

• GPT‐4 through o1 models achieve pooled accuracy of 87% in the anatomical sciences and ≥83% in five of six biomedical disciplines, consistently above older versions — related
• Newer OpenAI LLM versions achieve significantly higher pooled accuracy on biomedical MCQs than earlier versions (GPT‐4o–o1: 90%; GPT‐4: 82%; GPT‐3/3.5: 59%) — a broader claim this one bears on
• Question type, stem type, MCQ source, and web accessibility did not independently explain LLM accuracy heterogeneity in biomedical MCQ meta‐regressions — related
• Newer Anthropic and DeepSeek models approach GPT‐4o–o1 accuracy on biomedical MCQs, while Google Gemini and Microsoft Copilot trail the leading companies — related
• LLM accuracy does not differ significantly across question languages, ranging from 62% in Polish to 77% in German — related