ChatGPT performed at or near the passing threshold on the United States Medical Licensing Exam without domain fine-tuning (reported from Kung et al.)¶

Claim · All claims
Evidence · 1 study · theoretical r2 · q2
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person · recheck by 2026: tool and model accuracy changes quickly

Subclaims¶

q2 i? ChatGPT's performance on the USMLE was at or near the passing threshold without any domain fine-tuning, suggesting potential to assist medical education. → Kasneci 2023

Evidence¶

Kasneci 2023¶

Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., Stadler, M., Weller, J., Kuhn, J., & Kasneci, G. (2023). ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education. https://doi.org/10.1016/j.lindif.2023.102274

q2 · i? · theoretical · r2

Narrative review attribution in Section 2.2: the paper reports Kung et al.'s evaluation of ChatGPT on the United States Medical Licensing Exam, stating performance was "at or near the passing threshold without any domain fine-tuning"; no scores are printed in this review.

"According to the evaluation results, the performance of ChatGPT on this test was at or near the passing threshold without any domain fine-tuning."

Discussion¶

Related Claims¶

• Epi2Diff achieves the best performance across all four item-difficulty test sets and all reported metrics, outperforming SLM fine-tuning, LLM prompting, and supervised LLM adaptation baselines — related
• Task-specific fine-tuning substantially improves open-source LLMs on MedGame Bench and narrows the gap with commercial models — related
• Off-the-shelf LLMs without domain-specific fine-tuning suggested true root causes among their zero-shot debugging recommendations for buggy analog circuits — related
• AI assistance raises the probability of correctly identifying the task's root cause for both education groups, with near-equalization among treated participants — related