Editing questions according to a hypothesis shifts measured difficulty in the expected direction¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Hypothesis-guided increase edits reduce mean accuracy by 15.27 and 23.88 percentage points on GSM8K and BBH-structured, while decrease edits improve it by 32.18 and 28.52 points, with directional consistency across most edited questions. → Peng Cui 2026

Evidence¶

Peng Cui 2026¶

Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mrinmaya Sachan. (2026). What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language. Preprint. https://arxiv.org/abs/2610.01627 doi:10.48550/arxiv.2610.01627

q2 · i? · causal · r1

Counterfactual editing experiment (Table 4) on 50 randomly sampled test questions per dataset, re-evaluated on a subset of LLMs under 30B with accuracy as a difficulty proxy. The article reports the printed accuracy shifts and success rates of 70.9% and 92.2% on GSM8K.

"Increase-difficulty edits reduce mean accuracy by 15.27 and 23.88 percentage points on GSM8K and BBH-structured, respectively, while decrease-difficulty edits improve it by 32.18 and 28.52 points."

Discussion¶

Related Claims¶

• Generated hypotheses discriminate hard from easy questions, mostly with medium-to-large effects — related
• Hypothesis-based predictors match or beat black-box difficulty regressors on unseen questions — related
• The benefit of high- versus low-contrast sampling depends on the dataset — related
• Mixing group-wise and pair-wise contrastive sampling yields better hypotheses than either alone — related
• Adding generated hypotheses as features improves black-box difficulty predictors — related
• LLM difficulty labels track structural question features: stem length, option length, and digits in stems rise monotonically across Easy, Medium, and Hard labels — related
• Success rates decrease monotonically from easy to medium to hard difficulty levels across all models — related