LLM difficulty labels track structural question features: stem length, option length, and digits in stems rise monotonically across Easy, Medium, and Hard labels¶
Claim · All claims
Evidence · 1 study · associationalr1·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Structural features increased systematically across the three difficulty labels, while other surface characteristics did not separate levels. → Yuan An 2026
Evidence¶
Yuan An 2026¶
Yuan An, Lei Wang. (2026). Student Use of LLMs and the Limits of AI-Generated Question Difficulty in Data Science Courses. https://arxiv.org/abs/2609.27063 doi:10.48550/arxiv.2609.27063
q2 · i? · associational · r1
Stage 1 analysis of the 378 authored items (126 per label) showed mean stem length rising from 15.9 to 30.2 words and digit-containing stems from 5.6% to 26.2% across Easy, Medium, and Hard labels, per Figure 6.
"Question stem length, option length, and the presence of numerical digits increased systematically across Easy, Medium, and Hard categories (Fig-ure 6)."
Discussion¶
Related Claims¶
- LLM-assigned difficulty labels fail to predict empirical item difficulty in classroom retrieval practice (ρ=0.06), as do co-generated Bloom levels (ρ=0.02) — related
- Generated hypotheses discriminate hard from easy questions, mostly with medium-to-large effects — related
- Across frontier LLMs, Counseling is the easiest educational activity center (average 8.78) while Assessment and Research are hardest (8.42), and performance degrades from Easy to Hard difficulty — related
- Editing questions according to a hypothesis shifts measured difficulty in the expected direction — related
- Hypothesis-based predictors match or beat black-box difficulty regressors on unseen questions — related
- Adding generated hypotheses as features improves black-box difficulty predictors — related
- Success rates decrease monotonically from easy to medium to hard difficulty levels across all models — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
llm-difficulty-labels-track-structural-item-features |
title |
LLM difficulty labels track structural question features: stem length, option length, and digits in stems rise monotonically across Easy, Medium, and Hard labels |
description |
LLM difficulty labels track structural question features: stem length, option length, and digits in stems rise monotonically across Easy, Medium, and Hard labels |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — yuan-an-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.