Ground-truth difficulty labels in reviewed studies come roughly equally from expert/annotator labelling (34%), Classical Test Theory (34%), and Item Response Theory (26%)¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Across the reviewed publications, expert/annotator labelling and CTT each account for 34% of ground-truth labelling methods, IRT for 26%, with Delta, Equated Delta and automatic labelling rare. → Samah AlKhuzaey 2023

Evidence¶

Samah AlKhuzaey 2023¶

Samah AlKhuzaey, Floriana Grasso, Terry R. Payne, Valentina Tamma. (2023). Text-based Question Difficulty Prediction: A Systematic Review of Automatic Approaches. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00362-1

q2 · i? · review · r2

The review's synthesis of ground-truth labelling methods (its Table 4) reports expert/annotator labelling at 34% (19 papers), CTT at 34% (19), IRT at 26% (15), and Delta, Equated Delta and automatic labelling at 2% each. The quote gives the printed percentages for the two psychometric theories.

"The proportion correct statistic (CTT) and the Rasch model (IRT) were utilised in 34% and 26% of the studies respectively."

Discussion¶

Related Claims¶

• All surveyed automatic question difficulty prediction studies frame the task as supervised learning, mostly as classification of pre-defined difficulty categories — a broader claim this one bears on
• CTT and MFRM yield similar item difficulty results for open-ended questions regardless of crossed or nested design — related
• Learned difficulty parameters show a moderate positive correlation with empirical KC difficulty on QATD2k — related