Generated hypotheses discriminate hard from easy questions, mostly with medium-to-large effects¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? On GSM8K and BBH-structured, 9/10 top hypotheses reach at least a medium effect separating matched from unmatched items, and half reach a large effect; on Winogrande the effects are weaker. → Peng Cui 2026

Evidence¶

Peng Cui 2026¶

Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mrinmaya Sachan. (2026). What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language. Preprint. https://arxiv.org/abs/2610.01627 doi:10.48550/arxiv.2610.01627

q2 · i? · causal · r1

Numerical evaluation of top hypotheses on three benchmarks (Table 1), reporting SEP with 95% bootstrap CIs and Cohen's d between matched and unmatched test items. The article reports "the 95% CI of SEP has a lower bound above zero for 13/15 hypotheses" and medium-to-large effects outside WinoGrande.

"the 95% CI of SEP has a lower bound above zero for 13/15 hypotheses, showing that they genuinely discriminate between hard and easy items. Except on Winogrande, whose difficulty is known to be hard to perceive even for humans (Ding et al., 2024), 9/10 hypotheses on the two other datasets reach at least a medium effect†, and half of them reach a large effect‡."

Discussion¶

Related Claims¶

• Hypothesis-based predictors match or beat black-box difficulty regressors on unseen questions — related
• Mixing group-wise and pair-wise contrastive sampling yields better hypotheses than either alone — related
• The benefit of high- versus low-contrast sampling depends on the dataset — related
• Editing questions according to a hypothesis shifts measured difficulty in the expected direction — related
• Adding generated hypotheses as features improves black-box difficulty predictors — related
• LLM difficulty labels track structural question features: stem length, option length, and digits in stems rise monotonically across Easy, Medium, and Hard labels — related
• Success rates decrease monotonically from easy to medium to hard difficulty levels across all models — related