Fine-tuning LLMs and MLMs with cross-entropy loss using soft targets outperforms standard discretization and regression-head approaches on vocabulary difficulty prediction¶

Claim · All claims
Evidence · 1 study · causal r? · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Ablation results show that fine-tuning with soft-target cross-entropy loss yields lower RMSE than standard cross-entropy loss with discretized scores for both LLMs and MLMs, and than a regression head with MSE loss for MLMs. → Adam Nohejl 2026

Evidence¶

Adam Nohejl 2026¶

Adam Nohejl, Xuanxin Wu, Yusuke Ide, Maria Angelica Riera Machin, Yi-Ning Chang, Hitomi Yanaka. (2026). Sakura at BEA 2026 Shared Task 1: What Makes Vocabulary Difficult? https://arxiv.org/abs/2605.14257 doi:10.48550/arxiv.2605.14257

q2 · i? · causal · r?

Ablation analysis (Section 6.1, Tables 4 and 5) of the fine-tuned Ministral-3-14B LLM and mmBERT-base MLM used in the submissions, comparing the soft-target method against standard-loss and regression-head variants; the article reports a tangible performance drop for the standard loss ablations. The authors state the soft-target technique "is superior to common approaches for both LLMs and MLMs."

"The standard loss ablation demonstrates that our technique of fine-tuning using cross-entropy loss with soft targets is superior to common approaches for both LLMs and MLMs."

Discussion¶

Related Claims¶

• The team's open-track submissions with added features surpassed all other shared task submissions on all three languages, with RMSE near 0.7 — related
• In the fine-tuning experiments, increasing base model size up to 32B parameters improved performance, and larger LLMs consistently outperformed smaller MLMs — related
• A fine-tuned Qwen3.5-9B simulated-respondent model predicts item difficulty with Pearson correlation 0.85, exceeding MathBERT (0.68) and MetaMath (0.75) baselines — related
• Under a five-band discretization of difficulty, the simulated-respondent model reaches quadratic weighted kappa 0.835 versus 0.692 (MetaMath) and 0.625 (MathBERT) — related
• KL-divergence and binary cross-entropy losses for course prediction yield grade prediction within 1-2% of each other — related
• Discrimination parameter recovery is weak (Pearson 0.31 for Qwen3.5-9B, below MathBERT's 0.35), with slope information partially lost through ability discretization — related