Grader errors dominate audited rejections on benchmarks using rule-based evaluators (PHYBench and PRISM-Physics)¶

Claim · All claims
Evidence · 1 study (2 entries) · design r? · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? On PHYBench's audit set, 40 of 56 rejected questions are grader errors, versus 13 benchmark errors and 3 model errors. → Ali Ansari 2026 q2 i? On PRISM-Physics's audit set, 48 of 74 rejections are grader errors and none are model errors. → Ali Ansari 2026 (2)

Evidence¶

Ali Ansari 2026¶

Ali Ansari, Haoran Sun, Andy Zeyi Liu, Mark Jabbour, Yongshan Ding, Steven Girvin, Yu He, Sohrab Ismail-Beigi, Aleksander Kubica, Owen D. Miller, Corey O'Hern, Vidvuds Ozolins, David Poland, A. Douglas Stone, Frank C. van den Bosch, Logan Wright, Navid Akbari, Santanu Antu, Kangle Cai, Andrew Calabrese-Day, Mateo Cárdenes Wuttig, Meng Cheng, Barry T. Chiang, Ali Ghorashi, Shouzhen Gu, Haoyang Huang, Zhibo Kang, Lukas Kienesberger, Hantian Liu, Charles Lomba, Zhongling Lu, Wenchao Ma, Rohin E. McIntosh, Evan McKinney, Ivan Rojkov, Xulei Sun, Yarone Meir Tokayer, Naveen Balaji Umasankar, Mira Varma, Leda Wang, Qimin Wang, Tyler Wang, Haoyu Wei, Jinming Yang, Jinchen Zhao, Sherlock Tingrui Zhao, Qinyuan Zheng, Jay S. Zou, Lucas Baker, Arman Cohan, John Sous. (2026). How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks. https://arxiv.org/abs/2609.13009 doi:10.48550/arxiv.2609.13009

q2 · i? · design · r?

Expert audit of PHYBench rejections after up to five tool-free attempts on 100 answer-bearing questions. The article states the audit "attributes 13 to benchmark errors, 40 to grader errors, and 3 to model errors"; no effect size is printed.

"After up to five attempts, 56 questions remain rejected. The audit attributes 13 to benchmark errors, 40 to grader errors, and 3 to model errors."

Ali Ansari 2026 (2)¶

Ali Ansari, Haoran Sun, Andy Zeyi Liu, Mark Jabbour, Yongshan Ding, Steven Girvin, Yu He, Sohrab Ismail-Beigi, Aleksander Kubica, Owen D. Miller, Corey O'Hern, Vidvuds Ozolins, David Poland, A. Douglas Stone, Frank C. van den Bosch, Logan Wright, Navid Akbari, Santanu Antu, Kangle Cai, Andrew Calabrese-Day, Mateo Cárdenes Wuttig, Meng Cheng, Barry T. Chiang, Ali Ghorashi, Shouzhen Gu, Haoyang Huang, Zhibo Kang, Lukas Kienesberger, Hantian Liu, Charles Lomba, Zhongling Lu, Wenchao Ma, Rohin E. McIntosh, Evan McKinney, Ivan Rojkov, Xulei Sun, Yarone Meir Tokayer, Naveen Balaji Umasankar, Mira Varma, Leda Wang, Qimin Wang, Tyler Wang, Haoyu Wei, Jinming Yang, Jinchen Zhao, Sherlock Tingrui Zhao, Qinyuan Zheng, Jay S. Zou, Lucas Baker, Arman Cohan, John Sous. (2026). How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks. https://arxiv.org/abs/2609.13009

q2 · i? · design · r?

Expert audit of 74 rejected questions from a random 100-question sample of PRISM-Physics text-only problems. The article reports "48 to grader errors, and none to model errors"; no effect size is printed.

"The audit attributes 26 rejections to benchmark errors, 48 to grader errors, and none to model errors."

Discussion¶

Related Claims¶

• Expert audit attributes 95.20% of audited benchmark rejections to benchmark or grader errors rather than model errors — related
• On public-source benchmark subsets, corrected scores for all three frontier models rise dramatically after excluding flawed questions — related
• After expert correction of benchmark materials, GPT-5.6-Sol's measured accuracy rises substantially on HLE-Physics, CMT-Benchmark, and CritPt — related
• The HLE-adapted evaluator shows the lowest grader error rate (4.08%) among the evaluators audited — related