DAC reliability varies by category: Informatives (especially negative evaluations and nonsense statements) code poorly¶

Claim · All claims
Evidence · 1 study · design r3 · q3
Currency · newest study 2008 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Category kappas range from 0.04 to 1.00, and 5 of the 29 coding categories fall below the 0.70 agreement criterion, with three of these five in the Informatives category. → Erkens 2008

Evidence¶

Erkens 2008¶

Erkens, G., & Janssen, J. (2008). Automatic coding of dialogue acts in collaboration protocols. Computer-Supported Collaborative Learning, 3, 447-470. https://doi.org/10.1007/s11412-008-9052-6

q3 · i? · design · r3

Reliability analysis on the same 1,000-message Study 2 protocol (Table 4). The article notes "Especially the negative evaluations (InfEvlNeg) and nonsense statements (InfStmNon) are considerably below the 0.70 criterion" and that students' chat abbreviations (e.g., "luvU") were often unrecognized.

"the category Kappa 's range from an unacceptable 0.04 (0.12 without InfStm?) to an excellent 1.00. When judging these kappa 's against Krippendorff 's standard of a minimum agreement of 0.70, 5 of the 29 coding categories fall short of this criterion"

Discussion¶

Related Claims¶

• Automatically coded dialogue acts correlate with manually coded collaborative activities as expected
• Automatically coded dialogue acts distinguish male and female students' online communication
• Automatic DAC coding of dialogue acts agrees substantially with manual coding (79.0% agreement, kappa 0.75) — related