Future research should use broader question formats, non-English training data, and LLMs beyond ChatGPT¶

Strategy · All strategies
Evidence · 3 claims (3 for) · 1 study (1 quant-synthesis), q3 · 0 of 1 report an effect size · 3 claims rest on one study

Description¶

The review's forward-looking recommendations call for expanding evidence beyond multiple-choice questions and ChatGPT. The authors state that "future research should explore the application of LLMs beyond ChatGPT to offer an up-to-date perspective on their efficacy in medical and health education", and recommend incorporating additional training data in languages other than English to improve performance.

Design Implications¶

Context¶

Requirements¶

• Studies must include open-ended or scenario-based question formats and specialized training data for different question types.

Constraints¶

• The current evidence base covers only multiple-choice questions and ChatGPT models; no grey literature was included and the protocol was not registered.

Target Learners¶

• medical and health profession students and professionals

Target Learning Goals¶

• evaluating AI chatbot utility in health education and clinical settings

Claims¶

• Included studies report ChatGPT-3.5 accuracy from 36% to 77% and GPT-4 accuracy from 64.4% to 100% [+W]
• ChatGPT models achieve a pooled accuracy of 70.1% on national health licensing examinations [+W]
• GPT-4 significantly outperforms ChatGPT-3.5 in accuracy on national health licensing examinations [+W]

Related Strategies¶

• Broaden instruments, methods, and expert judgment in psychometric LLM evaluation research
• Standardize reporting of LLM versions, processing dates, input selection criteria, and data quality protocols in future MCQ benchmarking research
• Future research on professional reasoning in occupational therapy should target information processing, reasoning modalities, methodological quality assessment, and non-English-speaking contexts

Examples¶

-

Key Sources¶

• Jin Hye Kyung, Lee Ha Eun, Kim EunYoung. (2024). Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Medical Education. https://doi.org/10.1186/s12909-024-05944-8