A fine-tuned Qwen3.5-9B simulated-respondent model predicts item difficulty with Pearson correlation 0.85, exceeding MathBERT (0.68) and MetaMath (0.75) baselines¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? On a held-out test set of mathematics items, the fine-tuned Qwen3.5-9B model attains a Pearson correlation of 0.85 on the difficulty parameter b, with RMSE 0.55 after a linear development-set correction, exceeding both regression baselines. → Christopher Ormerod and YoungKoung Kim 2026

Evidence¶

Christopher Ormerod and YoungKoung Kim 2026¶

Christopher Ormerod and YoungKoung Kim. (2026). Multimodal Item Parameter Estimation using Simulated Response Probabilities. https://arxiv.org/abs/2608.10154 doi:10.48550/arxiv.2608.10154

q2 · i? · design · r2

Numerical results on a held-out test set (Table 5) comparing the proposed simulated-respondent fine-tuned LLM against encoder-only regression baselines; the article reports "a relative improvement of roughly 15% over the stronger baseline, obtained with an RMSE of0.55".

"The fine-tuned Qwen3.5-9B model attains a Pearson correlation of0.85 on the difficulty parameterb, exceeding both regression baselines—MathBERT (0.68) and MetaMath ( 0.75)—by a clear margin."

Discussion¶

Related Claims¶

• Discrimination parameter recovery is weak (Pearson 0.31 for Qwen3.5-9B, below MathBERT's 0.35), with slope information partially lost through ability discretization — related
• The simulated-respondent approach recovers the guessing parameter c at correlation 0.48, far above regression baselines, suggesting it captures genuine response behavior — related
• Under a five-band discretization of difficulty, the simulated-respondent model reaches quadratic weighted kappa 0.835 versus 0.692 (MetaMath) and 0.625 (MathBERT) — related
• Fine-tuning LLMs and MLMs with cross-entropy loss using soft targets outperforms standard discretization and regression-head approaches on vocabulary difficulty prediction — related
• In the fine-tuning experiments, increasing base model size up to 32B parameters improved performance, and larger LLMs consistently outperformed smaller MLMs — related
• Epi2Diff achieves the best performance across all four item-difficulty test sets and all reported metrics, outperforming SLM fine-tuning, LLM prompting, and supervised LLM adaptation baselines — related
• The LLM observation function's per-answer mastery evidence correlates with true mastery at r = 0.68 pooled, but only r ≈ 0.15 within the weak tier, making it least reliable for low-ability learners — related