On public-source benchmark subsets, corrected scores for all three frontier models rise dramatically after excluding flawed questions¶
Claim · All claims
Evidence · 1 study · causalr?·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Fable 5 High's pre-audit and corrected mean@4 are 39.50% and 87.64% on PHYBench, 7.50% and 84.80% on PRISM-Physics, and 78.25% and 87.80% on UGPhysics. → Ali Ansari 2026
Evidence¶
Ali Ansari 2026¶
Ali Ansari, Haoran Sun, Andy Zeyi Liu, Mark Jabbour, Yongshan Ding, Steven Girvin, Yu He, Sohrab Ismail-Beigi, Aleksander Kubica, Owen D. Miller, Corey O'Hern, Vidvuds Ozolins, David Poland, A. Douglas Stone, Frank C. van den Bosch, Logan Wright, Navid Akbari, Santanu Antu, Kangle Cai, Andrew Calabrese-Day, Mateo Cárdenes Wuttig, Meng Cheng, Barry T. Chiang, Ali Ghorashi, Shouzhen Gu, Haoyang Huang, Zhibo Kang, Lukas Kienesberger, Hantian Liu, Charles Lomba, Zhongling Lu, Wenchao Ma, Rohin E. McIntosh, Evan McKinney, Ivan Rojkov, Xulei Sun, Yarone Meir Tokayer, Naveen Balaji Umasankar, Mira Varma, Leda Wang, Qimin Wang, Tyler Wang, Haoyu Wei, Jinming Yang, Jinchen Zhao, Sherlock Tingrui Zhao, Qinyuan Zheng, Jay S. Zou, Lucas Baker, Arman Cohan, John Sous. (2026). How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks. https://arxiv.org/abs/2609.13009 doi:10.48550/arxiv.2609.13009
q2 · i? · causal · r?
Pre-audit versus corrected evaluation on retained subsets of three public-source benchmarks after excluding flawed questions. The article prints the paired scores verbatim, e.g. "7.50% and 84.80% on PRISM-Physics"; no effect size is reported.
"On the evaluated public-source subsets, Fable 5 High's pre-audit and corrected mean@4 are 39.50% and 87.64% on PHYBench, 7.50% and 84.80% on PRISM-Physics, and 78.25% and 87.80% on UGPhysics."
Discussion¶
Related Claims¶
- After expert correction of benchmark materials, GPT-5.6-Sol's measured accuracy rises substantially on HLE-Physics, CMT-Benchmark, and CritPt — related
- Expert audit attributes 95.20% of audited benchmark rejections to benchmark or grader errors rather than model errors — related
- Grader errors dominate audited rejections on benchmarks using rule-based evaluators (PHYBench and PRISM-Physics) — related
- The HLE-adapted evaluator shows the lowest grader error rate (4.08%) among the evaluators audited
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
public-source-benchmark-corrected-scores-rise-dramatically |
title |
On public-source benchmark subsets, corrected scores for all three frontier models rise dramatically after excluding flawed questions |
description |
On public-source benchmark subsets, corrected scores for all three frontier models rise dramatically after excluding flawed questions |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — ali-ansari-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.