After expert correction of benchmark materials, GPT-5.6-Sol's measured accuracy rises substantially on HLE-Physics, CMT-Benchmark, and CritPt¶

Claim · All claims
Evidence · 1 study (2 entries) · design r? · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? GPT-5.6-Sol's mean@4 rises from 47.3% to 78.7% on HLE-Physics and from 61.0% to 87.2% on CMT-Benchmark after correcting evaluation errors and repairing or excluding flawed questions. → Ali Ansari 2026 q2 i? On the 54 retained CritPt challenges, GPT-5.6-Sol's corrected pass@4 reaches 94.4% after expert repair and independent derivation of reference solutions. → Ali Ansari 2026 (2)

Evidence¶

Ali Ansari 2026¶

Ali Ansari, Haoran Sun, Andy Zeyi Liu, Mark Jabbour, Yongshan Ding, Steven Girvin, Yu He, Sohrab Ismail-Beigi, Aleksander Kubica, Owen D. Miller, Corey O'Hern, Vidvuds Ozolins, David Poland, A. Douglas Stone, Frank C. van den Bosch, Logan Wright, Navid Akbari, Santanu Antu, Kangle Cai, Andrew Calabrese-Day, Mateo Cárdenes Wuttig, Meng Cheng, Barry T. Chiang, Ali Ghorashi, Shouzhen Gu, Haoyang Huang, Zhibo Kang, Lukas Kienesberger, Hantian Liu, Charles Lomba, Zhongling Lu, Wenchao Ma, Rohin E. McIntosh, Evan McKinney, Ivan Rojkov, Xulei Sun, Yarone Meir Tokayer, Naveen Balaji Umasankar, Mira Varma, Leda Wang, Qimin Wang, Tyler Wang, Haoyu Wei, Jinming Yang, Jinchen Zhao, Sherlock Tingrui Zhao, Qinyuan Zheng, Jay S. Zou, Lucas Baker, Arman Cohan, John Sous. (2026). How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks. https://arxiv.org/abs/2609.13009 doi:10.48550/arxiv.2609.13009

q2 · i? · design · r?

Pre-audit versus validated/repaired evaluation of GPT-5.6-Sol on expert-authored benchmarks, with corrected scores computed on retained subsets after expert review. The article reports the rises verbatim: "mean@4 rises from 47.3% to 78.7%" on HLE-Physics and "from 61.0% to 87.2%" on CMT-Benchmark; no effect size is printed.

"On HLE-Physics, GPT-5.6-Sol's mean@4 rises from 47.3% to 78.7%. On CMT-Benchmark (Pan et al., 2026), it rises from 61.0% to 87.2%. On CritPt, the mean@5 over the 70 evaluated challenges is 32.3% before the audit, and the mean@4 over the 54 retained challenges is 87.5% after it, with a pass@4 of 94.4%."

Ali Ansari 2026 (2)¶

Ali Ansari, Haoran Sun, Andy Zeyi Liu, Mark Jabbour, Yongshan Ding, Steven Girvin, Yu He, Sohrab Ismail-Beigi, Aleksander Kubica, Owen D. Miller, Corey O'Hern, Vidvuds Ozolins, David Poland, A. Douglas Stone, Frank C. van den Bosch, Logan Wright, Navid Akbari, Santanu Antu, Kangle Cai, Andrew Calabrese-Day, Mateo Cárdenes Wuttig, Meng Cheng, Barry T. Chiang, Ali Ghorashi, Shouzhen Gu, Haoyang Huang, Zhibo Kang, Lukas Kienesberger, Hantian Liu, Charles Lomba, Zhongling Lu, Wenchao Ma, Rohin E. McIntosh, Evan McKinney, Ivan Rojkov, Xulei Sun, Yarone Meir Tokayer, Naveen Balaji Umasankar, Mira Varma, Leda Wang, Qimin Wang, Tyler Wang, Haoyu Wei, Jinming Yang, Jinchen Zhao, Sherlock Tingrui Zhao, Qinyuan Zheng, Jay S. Zou, Lucas Baker, Arman Cohan, John Sous. (2026). How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks. https://arxiv.org/abs/2609.13009

q2 · i? · design · r?

CritPt corrected evaluation using reference solutions independently derived by auditors because official solutions were unavailable. The article reports "corrected mean@4 reaches 87.50% for GPT-5.6-Sol Max and 78.24% for Fable 5 High"; pre-audit pass@4 was unavailable.

"On the 54 challenges retained from the 56 audited, corrected mean@4 reaches 87.50% for GPT-5.6-Sol Max and 78.24% for Fable 5 High, while corrected pass@4 reaches 94.44% and 90.74%, respectively."

Discussion¶

Related Claims¶

• On public-source benchmark subsets, corrected scores for all three frontier models rise dramatically after excluding flawed questions — related
• Expert audit attributes 95.20% of audited benchmark rejections to benchmark or grader errors rather than model errors — a broader claim this one bears on
• Grader errors dominate audited rejections on benchmarks using rule-based evaluators (PHYBench and PRISM-Physics) — related
• The HLE-adapted evaluator shows the lowest grader error rate (4.08%) among the evaluators audited — related