Hypothesis-based predictors match or beat black-box difficulty regressors on unseen questions¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? An OLS regressor on hypothesis-match features achieves the best out-of-sample R² on GSM8K and WinoGrande and second best on BBH-structured against finetuned, embedding-based, and few-shot baselines. → Peng Cui 2026

Evidence¶

Peng Cui 2026¶

Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mrinmaya Sachan. (2026). What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language. Preprint. https://arxiv.org/abs/2610.01627 doi:10.48550/arxiv.2610.01627

q2 · i? · causal · r1

Out-of-sample evaluation (Table 2): an OLS regressor trained on G∪V with hypothesis-match features is tested on T against RoBERTa-base finetuned, frozen embeddings, and 10-shot prompting of frontier LLMs. The article reports "the best performance on two datasets, GSM8K and WinoGrande".

"Overall, our model achieves the best performance on two datasets, GSM8K and WinoGrande, and the second best on BBH-structured, showing that these features alone already constitute a strong difficulty predictor."

Discussion¶

Related Claims¶

• Generated hypotheses discriminate hard from easy questions, mostly with medium-to-large effects — related
• Adding generated hypotheses as features improves black-box difficulty predictors — related
• Editing questions according to a hypothesis shifts measured difficulty in the expected direction — related
• LLM difficulty labels track structural question features: stem length, option length, and digits in stems rise monotonically across Easy, Medium, and Hard labels — related
• Baseline hypothesis generators saturate after 5–10 effective hypotheses while the proposed method keeps improving — related
• Direct zero-shot GPT-4o difficulty ratings beat a grade-level dummy regressor and correlate strongly with true Rasch difficulties overall, for K-5 math and reading items — related