Removing high-risk-of-bias studies does not materially change the pooled accuracy estimate¶

Claim · All claims
Evidence · 1 study · quant-synthesis r2 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? A sensitivity analysis excluding high-risk studies left overall accuracy at 71.3%, indicating robustness of the pooled result. → Jin Hye Kyung 2024

Evidence¶

Jin Hye Kyung 2024¶

Jin Hye Kyung, Lee Ha Eun, Kim EunYoung. (2024). Performance of ChatGPT-3.5 and GPT-4 in national licensing examinations for medicine, pharmacy, dentistry, and nursing: a systematic review and meta-analysis. BMC Medical Education. https://doi.org/10.1186/s12909-024-05944-8

q3 · i? · quant-synthesis · r2

Sensitivity analysis in the random-effects meta-analysis excluding the two studies judged at high risk of bias on the JBI checklist. Overall accuracy "remained at 71.3% (95% CI, 66.3-75.8%)" with I² of 98.1%.

"The sensitivity analysis demonstrated that removing high-risk studies did not significantly alter the overall accuracy, which remained at 71.3% (95% CI, 66.3-75.8%), indicating that the results were robust (p < 0.001), with an I² of 98.1%"

Discussion¶

Related Claims¶

• ChatGPT accuracy on health licensing examinations is highest in pharmacy and lowest in nursing — related
• Pooled gamification effects remain stable when high-risk-of-bias studies are excluded and in leave-one-out and fixed-effect sensitivity analyses — possibly the same claim (merge candidate)