A structured negotiated-coding process reached nearly 98% interrater agreement before machine learning¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Two coders coded all posts in blocks of 150 with negotiation after each block, and the final interrater agreement for coding was excellent, with nearly 98%. → Dornauer 2024

Evidence¶

Dornauer 2024¶

Dornauer, V., Netzer, M., Kaczkó, É., Norz, L.-M., & Ammenwerth, E. (2024). Automatic Classification of Online Discussions and Other Learning Traces to Detect Cognitive Presence. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-023-00335-4

q2 · i? · design · r2

Manual coding procedure using MAXQDA with a German CoI codebook: two coders first coded 50 posts together, then blocks of 150 separately, resolving differences by discussion and a third reviewer. Initial agreement was low at 30% before negotiation. The approach follows "negotiated coding" (Garrison et al., 2006).

"After this consensus process, the final interrater agreement for coding was excellent, with nearly 98%."

Discussion¶

Related Claims¶

• The authors produced the first fully CoI-coded German dataset of 1,521 meaningful units from 1,147 posts, available on request — related
• Three expert coders reached 77.15% overall agreement (Fleiss' κ = 0.763) on the LCT MOOC sample, and over half of their disagreements lay between Exploration and Integration — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• The LoU Interview shows high interrater reliability, and the rating procedure was streamlined over two years without loss of agreement — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• Rubric-based assessment showed substantial to almost-perfect inter-rater reliability across all three time points — related
• Machine learning detects five of eight reflective-writing categories with substantial or almost perfect reliability and three with moderate reliability (Cohen's κ .53–.85) — related