Translating non-English examination questions into English improved GPT-3.5's performance but had limited effect on GPT-4¶

Claim · All claims
Evidence · 1 study · quant-synthesis r3 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q2 i? In 8 comparative studies, GPT-4's accuracy improvement from English translation ranged from 0.17% to 8.65%, while GPT-3.5 showed significant improvement in 4 studies, exceeding 20% in 2 of them. → Mingxin Liu 2024

Evidence¶

Mingxin Liu 2024¶

Mingxin Liu, Tsuyoshi Okuhara, XinYi Chang, Ritsuko Shirabe, Yuriko Nishiie, Hiroko Okada, Takahiro Kiuchi. (2024). Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis. Journal of Medical Internet Research. https://doi.org/10.2196/60807

q2 · i? · quant-synthesis · r3

Qualitative comparison of 9 studies that tested both original-language and English-translated questions, with 8 reporting comparative results (Table 2). The review reports GPT-4's "accuracy improvement ranged from 0.17% to 8.65%", while GPT-3.5's accuracy was more than 20% higher in English in 2 studies.

"Overall, for GPT-4, translating the original language into English had a limited effect on improving the performance. The accuracy improvement ranged from 0.17% to 8.65%, with 6 studies showing an accuracy increase of less than 5%."

Discussion¶

Related Claims¶

• LLM accuracy does not differ significantly across question languages, ranging from 62% in Polish to 77% in German — related