CTT and MFRM yield similar item difficulty results for open-ended questions regardless of crossed or nested design¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2018 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2021: tool and model accuracy changes quickly

Subclaims¶

q2 i? Combining this study's crossed design with Huang et al.'s (2014) nested design, the authors infer CTT and MFRM agree on item difficulty for open-ended questions under either design. → Ilhan 2018

Evidence¶

Ilhan 2018¶

Ilhan, M., & Guler, N. (2018). A Comparison of Difficulty Indices Calculated for Open-Ended Items According to Classical Test Theory and Many Facet Rasch Model. Eurasian Journal of Educational Research, 75. doi:10.14689/ejer.2018.75.6

q2 · i? · associational · r2

Authors' discussion-level inference combining this crossed-design study with the nested-design study of Huang et al. (2014), which the review-style discussion describes as rating 124 grant applications with 64 experts. This is the authors' interpretation, not a single tested contrast.

"Therefore, upon considering the results obtained in Huang et al. (2014) and th ose obtained in this study, the CTT and the MFRM yielded similar results in terms of item difficulty indices for open -ended questions, no matter which design – crossed or nested – was used."

Discussion¶

Related Claims¶

• CTT and MFRM yield a positive, perfect correlation between item difficulty indices for open-ended items under a crossed design — a narrower finding that bears on this claim
• Items ranked from easiest to most difficult are identical under CTT and MFRM — a narrower finding that bears on this claim
• Raters differed significantly in severity/leniency in the MFRM analysis — related
• Scorer severity and standard difficulty were statistically adjusted using Many-Facet Rasch analysis before assignment and work measures were compared — related
• CTT-based quiz difficulty rankings are validated by IRT analysis with near-perfect rank correlation — related
• Australian and Malaysian Muslim learners ranked discord items differently, with Australian students most resistant to anti-science statements and most accepting of divine-cause statements — related