Standardize reporting of LLM versions, processing dates, input selection criteria, and data quality protocols in future MCQ benchmarking research¶

Strategy · All strategies
Evidence · 4 claims (4 for) · 1 study (1 quant-synthesis), q4 · 0 of 1 report an effect size · 4 claims rest on one study

Description¶

In its future directions, the article calls for standardized reporting in primary research, akin to the CONSORT‐AI guidelines for clinical trials, noting that "Standardized reporting of LLM versions, processing dates, input selection criteria, and data quality protocols could minimize study ambiguities/heterogeneity and enhance study replicability." Authors should explicitly state how MCQs were developed or obtained and whether they were accessible online, especially since newer LLMs possess web browsing capabilities. Comparative discipline‐specific assessments of non‐OpenAI models and of custom chatbots should also be broadened as data become available.

Design Implications¶

Context¶

Requirements¶

• Primary studies must document LLM versions, processing dates, how MCQs were obtained, and whether questions were web‐accessible

Constraints¶

• The article notes this stems from observed heterogeneity and lack of explanatory covariates aside from LLM version; some disciplines could not be analyzed independently due to inconsistent reporting

Target Learners¶

• researchers benchmarking LLM performance in biomedical education

Target Learning Goals¶

• reliable evaluation of LLM accuracy for educational use cases

Claims¶

• Question type, stem type, MCQ source, and web accessibility did not independently explain LLM accuracy heterogeneity in biomedical MCQ meta‐regressions [+W]
• LLM version progression explains about half of the variance in biomedical MCQ accuracy, with GPT‐4o_o1 over six times as likely as GPT‐3/3.5 to answer correctly [+W]
• Newer OpenAI LLM versions achieve significantly higher pooled accuracy on biomedical MCQs than earlier versions (GPT‐4o–o1: 90%; GPT‐4: 82%; GPT‐3/3.5: 59%) [+W]
• Newer Anthropic and DeepSeek models approach GPT‐4o–o1 accuracy on biomedical MCQs, while Google Gemini and Microsoft Copilot trail the leading companies [+W]

Related Strategies¶

• Establish standardized misconception benchmarks using isomorphic items, paraphrases, and multi-turn curricula
• Develop rigorous benchmarks for LLM data extraction tasks in education systematic reviews
• Extend educational LLM benchmarking toward multilingual contexts, real classroom interaction data, and targeted training for identified capability gaps
• Staged future-research agenda for transitioning LC assessment tools from developmental validation to safe clinical integration
• Future research should use broader question formats, non-English training data, and LLMs beyond ChatGPT

Examples¶

-

Key Sources¶

• Cheverko CM, Mavrych V, Bolgova O, Mohamed FRR, Westrick J, Juarez L, Rush E, Solka KA, Doubleday AF, Byram JN, Becker R, Gomez V, Ganeng BKA, Hoffman LA, Roach VA, Brown KM, DeVaul N, Garnett CN, Herriott HL, Lufler RS, Mussell JC, Balta JY, Pascoe MA, Middleton JW, Duffy S, Stephens GC, Wilson AB. (2026). The performance of ChatGPT and other large language models on multiple‐choice questions in biomedical disciplines: A meta‐analysis. Anatomical Sciences Education. https://doi.org/10.1002/ase.70262