LLMs struggle most with Chemistry riddles, with best EM accuracy at or below 75% versus above 90% in other subjects¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across all models, performance was strong in all subjects except Chemistry, where the best models scored equal to or below 75% EM accuracy while other subjects exceeded 90%. → George Boateng 2026

Evidence¶

George Boateng 2026¶

George Boateng, Naafi Dasana Ibrahim, Samuel John, Philemon Badu, Patrick Agyeman-Budu, Jonathan Abrefah Mensah, Kevin Takyi Yeboah, William Edor, Andrew Kojo Mensah-Onumah, Nana Sam Yeboah, and Victor Kumbol. (2026). NSMQ Riddles: A Benchmark of Scientific and Mathematical Riddles for Quizzing Large Language Models. arXiv preprint. https://arxiv.org/abs/2605.07051 doi:10.48550/arxiv.2605.07051

q2 · i? · design · r2

Subject breakdown of the offline evaluation on 2019 NSMQ riddles (Table 5). The article attributes the Chemistry difficulty to handling of organic compounds and complex terminology, with models oversimplifying answers, e.g. answering "alkane" instead of "Cyclohexane".

"Across all models, our evaluation indicated that they performed quite well across all subjects except Chemistry, which posed a significant challenge, with the best models performing equal to or below 75% EM accuracy compared to the other subjects, which had the best models above 90%."

Discussion¶

Related Claims¶

• The best LLMs perform below the best student contestants in the real-time proxy riddle round — related
• In offline evaluation on NSMQ Riddles, closed LLMs generally outperform open models and higher reasoning settings improve performance — related
• LLM performance drops from offline to real-time proxy evaluation when clues arrive incrementally — related
• LLM teaching effectiveness varies substantially across subject domains, strongest where knowledge points apply directly and weakest in physics and chemistry — related
• ITS produce significant positive effects at all grade levels and in almost all subject domains evaluated, with exceptions such as chemistry — related