Test item construction, including avoidance of cognate L1 equivalents, often affects KVL difficulty scores beyond the genuine difficulty of the words¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? A trickiness feature, operationalized as the probability of an LLM answering the test item incorrectly, reveals that the construction of test items often affects the resulting difficulty scores. → Adam Nohejl 2026

Evidence¶

Adam Nohejl 2026¶

Adam Nohejl, Xuanxin Wu, Yusuke Ide, Maria Angelica Riera Machin, Yi-Ning Chang, Hitomi Yanaka. (2026). Sakura at BEA 2026 Shared Task 1: What Makes Vocabulary Difficult? https://arxiv.org/abs/2605.14257 doi:10.48550/arxiv.2605.14257

q2 · i? · design · r2

Analysis of high-trickiness items (Section 6.3, Table 6) using GPT-4.1-mini response probabilities on L1 Spanish items such as instantly, motivational, and reform. The authors observe that KVL creators often intentionally avoided cognate L1 words, and the chosen L1 words strongly suggest different English words.

"it provides an interesting insight into how the test item construction often affects the resulting difficulty scores"

Discussion¶

Related Claims¶

• SHAP analysis of the explainable model identifies production frequency and spelling difficulty as high-impact predictors of KVL difficulty, with spelling difficulty varying strongly by L1 — related
• The simulated-respondent approach recovers the guessing parameter c at correlation 0.48, far above regression baselines, suggesting it captures genuine response behavior — related
• Incidental vocabulary learning through viewing TV was affected by frequency of occurrence, prior vocabulary knowledge, and cognateness — related
• LLM-extracted cognitive and linguistic features add predictive value beyond readily available item metadata (grade, domain, item type, word count) — related