Answering early on few clues correlates with worse real-time proxy performance¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Claude Opus 4.6's tendency to answer early correlates with its worse real-time proxy performance relative to other models, despite strong offline performance; GPT-OSS-120B behaves similarly. → George Boateng 2026

Evidence¶

George Boateng 2026¶

George Boateng, Naafi Dasana Ibrahim, Samuel John, Philemon Badu, Patrick Agyeman-Budu, Jonathan Abrefah Mensah, Kevin Takyi Yeboah, William Edor, Andrew Kojo Mensah-Onumah, Nana Sam Yeboah, and Victor Kumbol. (2026). NSMQ Riddles: A Benchmark of Scientific and Mathematical Riddles for Quizzing Large Language Models. arXiv preprint. https://arxiv.org/abs/2605.07051 doi:10.48550/arxiv.2605.07051

q2 · i? · design · r2

Behavioral analysis of the real-time proxy evaluation, computing the average number of clues each model waits for before answering. Claude Opus 4.6 answers earliest (avg clue correct 3.188 at high reasoning) and GPT-OSS-120B similarly early, while Kimi-K2.5 and DeepSeek-V3.1 wait for at least 4 clues on average.

"This tendency to answer early correlates with Claude Opus 4.6's worse performance in the real-time proxy test compared to the other models, despite its strong performance in the offline evaluation."

Discussion¶

Related Claims¶

• The best LLMs perform below the best student contestants in the real-time proxy riddle round — related
• LLM performance drops from offline to real-time proxy evaluation when clues arrive incrementally — a broader claim this one bears on