ML scoring models for scientific argumentation items achieved good to almost perfect agreement with human consensus scores (mean Cohen's kappa = 0.60)¶

Claim · All claims
Evidence · 1 study · associational r1 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? ML scoring models for 17 argumentation items yielded mean Cohen's kappa of 0.60 with human consensus scores, ranging from 0.38 to 0.89. → Haudek 2024

Evidence¶

Haudek 2024¶

Haudek, K. C., & Zhai, X. (2024). Examining the Effect of Assessment Construct Characteristics on Machine Learning Scoring of Scientific Argumentation. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00385-8

q2 · i? · associational · r1

ML models trained on 361 randomly selected responses per item using an ensemble bag-of-words approach with 10-fold cross-validation. Mean Cohen's kappa was "0.60" with a range of "0.38 − 0.89" across 17 items.

"The scoring models yielded a range of agreements with human consensus scores, measured by Cohen's kappa (mean = 0.60; range 0.38 − 0.89), indicating good to almost perfect performance."

Discussion¶

Related Claims¶

• Higher Complexity of assessment items is associated with decreased ML scoring model performance — related
• Machine learning detects five of eight reflective-writing categories with substantial or almost perfect reliability and three with moderate reliability (Cohen's κ .53–.85) — related
• A fine-tuned BERT model achieves almost perfect agreement with human coders when classifying student self-affirmation essays, matching human-human reliability — related
• NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward — a broader claim this one bears on
• The rule-based recipe annotation system reaches near-perfect inter-annotator agreement (Cohen's Kappa 0.85) and average accuracy 0.95 in a validation on sampled recipe segments. — related
• No significant differences in ML model performance across the three science item contexts — related
• No correlation between evenness of response score distribution and ML model performance — related