Build new frontier-model benchmarks from new hard tasks with expert verification and adequate curation resources¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article recommends replacing near-saturated, defect-ridden closed-ended benchmarks with a new style of evaluation. It states the field "calls for a new style of benchmark for frontier models, built from new hard physics tasks and adequately verified", noting such a benchmark "will likely require substantial financial resources to achieve high-quality curation". It suggests nonprofit organizations with rigorous processes or open competitions in the style recently adopted in mathematics as possible data-collection mechanisms.

Design Implications¶

Context¶

Requirements¶

• Substantial financial resources for high-quality curation
• Expert verification of tasks and reference solutions

Constraints¶

• The article notes this does not follow that frontier models are capable of end-to-end physics research; agents made considerably less progress on open physics problems than on mathematics

Target Learners¶

• benchmark developers
• AI evaluation organizations

Target Learning Goals¶

• objective measurement of frontier model physics capability

Related Strategies¶

• Create small controlled presentation mini-sets as a near-term community benchmark

Examples¶

-

Key Sources¶

• Ali Ansari, Haoran Sun, Andy Zeyi Liu, Mark Jabbour, Yongshan Ding, Steven Girvin, Yu He, Sohrab Ismail-Beigi, Aleksander Kubica, Owen D. Miller, Corey O'Hern, Vidvuds Ozolins, David Poland, A. Douglas Stone, Frank C. van den Bosch, Logan Wright, Navid Akbari, Santanu Antu, Kangle Cai, Andrew Calabrese-Day, Mateo Cárdenes Wuttig, Meng Cheng, Barry T. Chiang, Ali Ghorashi, Shouzhen Gu, Haoyang Huang, Zhibo Kang, Lukas Kienesberger, Hantian Liu, Charles Lomba, Zhongling Lu, Wenchao Ma, Rohin E. McIntosh, Evan McKinney, Ivan Rojkov, Xulei Sun, Yarone Meir Tokayer, Naveen Balaji Umasankar, Mira Varma, Leda Wang, Qimin Wang, Tyler Wang, Haoyu Wei, Jinming Yang, Jinchen Zhao, Sherlock Tingrui Zhao, Qinyuan Zheng, Jay S. Zou, Lucas Baker, Arman Cohan, John Sous. (2026). How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks. https://arxiv.org/abs/2609.13009