Adding generated hypotheses as features improves black-box difficulty predictors¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Supplying hypothesis-match vectors as additional features raises test R² of RoBERTa-base and frozen-embedding regressors, with the largest gains on GSM8K (+0.19, +0.18, and +0.11 over finetuned RoBERTa). → Peng Cui 2026

Evidence¶

Peng Cui 2026¶

Peng Cui, Qiaoyuan Zheng, Rudolf Debelak, Mrinmaya Sachan. (2026). What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language. Preprint. https://arxiv.org/abs/2610.01627 doi:10.48550/arxiv.2610.01627

q2 · i? · associational · r2

Feature-augmentation experiment (Figure 3): hypothesis-match vectors are added as a second feature block to three semantic predictors. The article reports gains "largest on GSM8K:+0.19and+0.18over the two frozen embeddings, and+0.11even over the fine-tuned RoBERTa".

"The gains are largest on GSM8K:+0.19and+0.18over the two frozen embeddings, and+0.11even over the fine-tuned RoBERTa (0.362→0.468), by far the strongest baseline there."

Discussion¶

Related Claims¶

• Baseline hypothesis generators saturate after 5–10 effective hypotheses while the proposed method keeps improving — related
• Generated hypotheses discriminate hard from easy questions, mostly with medium-to-large effects — related
• Hypothesis-based predictors match or beat black-box difficulty regressors on unseen questions — related
• Editing questions according to a hypothesis shifts measured difficulty in the expected direction — related
• LLM difficulty labels track structural question features: stem length, option length, and digits in stems rise monotonically across Easy, Medium, and Hard labels — related