Three expert coders reached 77.15% overall agreement (Fleiss' κ = 0.763) on the LCT MOOC sample, and over half of their disagreements lay between Exploration and Integration¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q3
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? After round-by-round training, the three expert coders reached an overall percentage agreement of 77.15% and Fleiss' κ of 0.763 on the sample of LCT MOOC discussion messages. → Yuanyuan Hu 2023 q3 i? Most coder disagreements involved two labels, with more than half labelled between Exploration and Integration, locating disagreements in the overlap between adjacent phases. → Yuanyuan Hu 2023 (2)

Evidence¶

Yuanyuan Hu 2023¶

Yuanyuan Hu. (2023). Can Multi-Label Classifiers Help Identify Subjectivity? A Deep Learning Approach to Classifying Cognitive Presence in MOOCs. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-022-00310-5

q3 · i? · design · r2

Manual categorisation study: three expert coders, trained in rounds of new messages until over 80% agreement without negotiations, classified the LCT MOOC sample into the five cognitive presence categories using an adapted coding rubric, reaching "an overall percentage agreement of 77.15% and Fleiss' κ of 0.763".

"Finally, they reached an overall percentage agreement of 77.15% and Fleiss' κ of 0.763."

Yuanyuan Hu 2023 (2)¶

Yuanyuan Hu. (2023). Can Multi-Label Classifiers Help Identify Subjectivity? A Deep Learning Approach to Classifying Cognitive Presence in MOOCs. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-022-00310-5

q3 · i? · design · r2

Analysis of the coder disagreements (Table 3, Fig. 3). The article reports "Most of the disa- greements had two labels, with more than half labelled between Exploration and Inte- gration by the coders", which the authors depict as messages sitting in overlapping areas between adjacent categories.

"Most of the disa- greements had two labels, with more than half labelled between Exploration and Inte- gration by the coders."

Discussion¶

Related Claims¶

• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• Automated judging aligns strongly with three domain experts (r = 0.82 for role fidelity) but exhibits a conservative bias, scoring ethical deviation 0.08 points lower than humans — related
• A structured negotiated-coding process reached nearly 98% interrater agreement before machine learning — related
• In the coded dataset, exploration was the most frequent phase (727 units, 48%), followed by integration (444, 29%), resolution (235, 15%), and non-cognitive presence (115, 8%) — related