Item difficulty was best predicted using signal from the item stem¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2020 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2023: tool and model accuracy changes quickly

Subclaims¶

q2 i? Among the item components tested as input, the item stem (the description of the clinical case) provided the best signal for predicting difficulty. → Kang Xue 2020

Evidence¶

Kang Xue 2020¶

Kang Xue, Victoria Yaneva, Christopher Runyon, Peter Baldwin. (2020). Predicting the difficulty and response time of multiple choice questions using transfer learning. 15th Workshop on Innovative Use of NLP for Building Educational Applications. https://www.nwea.org/research/publication/predicting-the-difficulty-and-response-time-of-multiple-choice-questions-using-transfer-learning/

q2 · i? · associational · r2

In the modeling study of the medical exam items, the authors report that "difficulty was best predicted using signal from the item stem (the description of theclinical case)"; no effect size is printed in the available text.

"In addition, difficulty was best predicted using signal from the item stem (the description of theclinical case), while all parts of the item were important for predicting the response time."

Discussion¶

Related Claims¶

• All parts of the item were important for predicting response time — related
• In Mnemosyne log data, item-specific difficulty parameters outperform a global difficulty for lower and higher Leitner decks, while global difficulty performs better for intermediate decks. — related
• Transfer learning improves prediction of item difficulty when response time is used as an auxiliary task, but not the other way around — related
• Combining semantic and process features yields the best difficulty prediction; semantic embeddings alone trail the full model by 7.4–12.1 macro-F1 points on SAT Math — related