No correlation between evenness of response score distribution and ML model performance¶

Claim · All claims
Evidence · 1 study · associational r2 · q2 · i0 negligible
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i0 There was no correlation between the balance of responses across rubric levels (evenness) and Cohen's kappa (Pearson's r = .08, p = .765, N = 17). → Haudek 2024

Evidence¶

Haudek 2024¶

Haudek, K. C., & Zhai, X. (2024). Examining the Effect of Assessment Construct Characteristics on Machine Learning Scoring of Scientific Argumentation. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00385-8

q2 · i0 · associational · r2

Pearson correlation across 17 items between evenness (Pielou diversity index of response score distribution) and Cohen's kappa. The article reports "r = .08, p = .765" indicating no relationship. A weak positive correlation with recall (r = .39, p = .118) was also non-significant.

"Surprisingly, there was no correlation between the balance of responses across rubric levels and Cohen's kappa of the resulting model (Pearson's r = .08, p = .765, N = 17)."

Discussion¶

Related Claims¶

• No significant differences in ML model performance across the three science item contexts — related
• Higher Structure (learning progression level) of assessment items is associated with decreased ML scoring model performance — related
• Higher Complexity of assessment items is associated with decreased ML scoring model performance — related
• ML scoring models for scientific argumentation items achieved good to almost perfect agreement with human consensus scores (mean Cohen's kappa = 0.60) — related