The best LLMs perform below the best student contestants in the real-time proxy riddle round¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? The best LLM performance of 75.64% EM accuracy and 366 points is lower than the best students' 78.21% EM accuracy and 377 points. → George Boateng 2026

Evidence¶

George Boateng 2026¶

George Boateng, Naafi Dasana Ibrahim, Samuel John, Philemon Badu, Patrick Agyeman-Budu, Jonathan Abrefah Mensah, Kevin Takyi Yeboah, William Edor, Andrew Kojo Mensah-Onumah, Nana Sam Yeboah, and Victor Kumbol. (2026). NSMQ Riddles: A Benchmark of Scientific and Mathematical Riddles for Quizzing Large Language Models. arXiv preprint. https://arxiv.org/abs/2605.07051 doi:10.48550/arxiv.2605.07051

q2 · i? · design · r2

Comparison of LLM real-time proxy results against the retrospective real-world performance of the best-performing student teams on the same 2019 riddles, using a metadata sheet recording which clue each school answered on. The article concludes the best LLMs currently perform below the smartest NSMQ students.

"The best performance of LLMs of 75.64% EM accuracy and 366 points is lower than the best students' performance of 78.21% EM accuracy and 377 points."

Discussion¶

Related Claims¶

• LLM performance drops from offline to real-time proxy evaluation when clues arrive incrementally — related
• Answering early on few clues correlates with worse real-time proxy performance — related
• In offline evaluation on NSMQ Riddles, closed LLMs generally outperform open models and higher reasoning settings improve performance — related
• LLMs struggle most with Chemistry riddles, with best EM accuracy at or below 75% versus above 90% in other subjects — related