Higher Structure (learning progression level) of assessment items is associated with decreased ML scoring model performance¶

Claim · All claims
Evidence · 1 study · associational r1 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? There was a significant negative correlation between item Structure level and Cohen's kappa (rs = −0.764, p < .001, N = 17). → Haudek 2024

Evidence¶

Haudek 2024¶

Haudek, K. C., & Zhai, X. (2024). Examining the Effect of Assessment Construct Characteristics on Machine Learning Scoring of Scientific Argumentation. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00385-8

q2 · i? · associational · r1

Spearman's rho correlation across 17 items between Structure level (alignment to learning progression levels) and Cohen's kappa. The article reports "rs = − 0.764, p < .001".

"There was a significant negative correlation between the two variables (rs = − 0.764, p < .001, N = 17), indicating the more complicated structure of the construct, the lower agreement between machine and human scores."

Discussion¶

Related Claims¶

• Higher Complexity of assessment items is associated with decreased ML scoring model performance — related
• No significant differences in ML model performance across the three science item contexts — related
• No correlation between evenness of response score distribution and ML model performance — related