No significant differences in ML model performance across the three science item contexts¶

Claim · All claims
Evidence · 1 study · associational r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? A Kruskal-Wallis H test found no significant differences in Cohen's kappa among the three item contexts (Chi-square = 3.55, p = .169, df = 2). → Haudek 2024

Evidence¶

Haudek 2024¶

Haudek, K. C., & Zhai, X. (2024). Examining the Effect of Assessment Construct Characteristics on Machine Learning Scoring of Scientific Argumentation. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00385-8

q2 · i? · associational · r2

Kruskal-Wallis H test comparing Cohen's kappa across sugar, bacteria, and gas item contexts. The article reports "Chi-square = 3.55, p = .169, df = 2". Despite this, gas items showed lower mean kappa (0.52) than sugar (0.65) or bacteria (0.69).

"No significant differences (Chi-square = 3.55, p = .169, df = 2) were found among the three item contexts."

Discussion¶

Related Claims¶

• Higher Complexity of assessment items is associated with decreased ML scoring model performance — related
• ML scoring models for scientific argumentation items achieved good to almost perfect agreement with human consensus scores (mean Cohen's kappa = 0.60) — related
• Higher Structure (learning progression level) of assessment items is associated with decreased ML scoring model performance — related
• No correlation between evenness of response score distribution and ML model performance — related