Learning-tip-to-lecture-topic coding by seven coders reached an overall agreement of 0.87 (Cohen's Kappa)¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person
Design limits · Gombert 2024: no comparison group (read from the study by two models, 2026-10-28)

Subclaims¶

q2 i? In the pilot phase for the multilabel topic coding, seven coders reached an overall Cohen's Kappa agreement of 0.87, and remaining essays were then split among coders. → Gombert 2024

Evidence¶

Gombert 2024¶

Gombert, S., Fink, A., Giorgashvili, T., Jivet, I., Di Mitri, D., Yau, J., Frey, A., & Drachsler, H. (2024). From the Automated Assessment of Student Essay Content to Highly Informative Feedback: a Case Study. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00387-6

q2 · i? · causal · r2

Manual annotation study on a shared Excel sheet with 100 columns (learning tip by lecture topic combinations): 100 pilot essays were coded in parallel by two of seven coders and Cohen's Kappa was computed. The article reports the coders "determine[d] the overall agreement (0.87)" and resolved disagreements by discussion.

"We then used these individual values to determine the overall agreement (0.87). After this step, fundamental disagreements found during coding were resolved through discussion."

Discussion¶

Related Claims¶

• Sentence-to-learning-tip segmentation coding by three trained coders reached an average Cohen's Kappa of 0.96 — related
• The segmentation training corpus covers 434 of 698 essays, yielding 22,581 coded sentences — related
• A fine-tuned BERT model achieves almost perfect agreement with human coders when classifying student self-affirmation essays, matching human-human reliability — related
• NLP models in LA studies typically reach moderate agreement (mean Cohen's kappa 0.54) and mean accuracy 0.79, with deep learning models outperforming others in most studies from 2021 onward — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• Machine learning detects five of eight reflective-writing categories with substantial or almost perfect reliability and three with moderate reliability (Cohen's κ .53–.85) — related
• A structured negotiated-coding process reached nearly 98% interrater agreement before machine learning — related