Higher Complexity of assessment items is associated with decreased ML scoring model performance¶

Claim · All claims
Evidence · 1 study · associational r1 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? There was a significant negative correlation between item Complexity level and Cohen's kappa (rs = −0.752, p < .001, N = 17). → Haudek 2024

Evidence¶

Haudek 2024¶

Haudek, K. C., & Zhai, X. (2024). Examining the Effect of Assessment Construct Characteristics on Machine Learning Scoring of Scientific Argumentation. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00385-8

q2 · i? · associational · r1

Spearman's rho correlation across 17 items between Complexity level and Cohen's kappa. The article reports "rs = − 0.752, p < .001" indicating more complex constructs yield lower human-machine agreement.

"There was a significant negative correlation between the two variables (rs = − 0.752, p < .001, N = 17), indicating that the more complex the construct, the lower amount of agreement between machine and human scores."

Discussion¶

Related Claims¶

• Higher Structure (learning progression level) of assessment items is associated with decreased ML scoring model performance — related
• ML scoring models for scientific argumentation items achieved good to almost perfect agreement with human consensus scores (mean Cohen's kappa = 0.60) — related
• No significant differences in ML model performance across the three science item contexts — related
• No correlation between evenness of response score distribution and ML model performance — related