Sentence-to-learning-tip segmentation coding by three trained coders reached an average Cohen's Kappa of 0.96¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Three coders annotating essay sentences with learning-tip codes achieved a global average Cohen's Kappa of 0.96, described as almost perfect agreement, so no further parallel coding was done. → Gombert 2024

Evidence¶

Gombert 2024¶

Gombert, S., Fink, A., Giorgashvili, T., Jivet, I., Di Mitri, D., Yau, J., Frey, A., & Drachsler, H. (2024). From the Automated Assessment of Student Essay Content to Highly Informative Feedback: a Case Study. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00387-6

q2 · i? · design · r2

Manual annotation study of the segmentation training data: three coders coded the same 100 pilot essays in INCEpTION and pairwise Cohen's Kappa was averaged. The article reports the global average was "determined to be 0.96", after which remaining essays were distributed between the coders.

"Then we calculated the global average which was determined to be 0.96, indicating an 'almost perfect' agreement between all three coders."

Discussion¶

Related Claims¶

• Learning-tip-to-lecture-topic coding by seven coders reached an overall agreement of 0.87 (Cohen's Kappa) — related
• NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward — a broader claim this one bears on
• The segmentation training corpus covers 434 of 698 essays, yielding 22,581 coded sentences — related