Automatic DAC coding of dialogue acts agrees substantially with manual coding (79.0% agreement, kappa 0.75)¶
Claim · All claims
Evidence · 1 study · designr3·q3
Currency · newest study 2008 · not yet checked against its sources · not reviewed by a person · overdue for a recheck since 2011: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q3 i? Comparing automatically coded dialogue acts to a human coder's coding of the same 1,000 chat messages yields 79.0% overall agreement and Cohen's kappa of 0.75 (85.6% and 0.84 when InfStm? codes are excluded). → Erkens 2008
Evidence¶
Erkens 2008¶
Erkens, G., & Janssen, J. (2008). Automatic coding of dialogue acts in collaboration protocols. Computer-Supported Collaborative Learning, 3, 447-470. https://doi.org/10.1007/s11412-008-9052-6
q3 · i? · design · r3
Reliability analysis comparing DAC-filter coding to a human coder familiar with the 29 dialogue acts on two random segments of 500 chat messages each from Study 2. The article reports "79.0%" agreement and kappa "0.75", which it calls "evidence for the reliability of the automatic coding procedure".
"The overall agreement between the automatic coding and the manual coding was found to be 79.0% (85.6% if the InfStm? codes are left out of the calculation), while Cohen 's kappa reached 0.75 (0.84 without InfStm?)."
Discussion¶
Related Claims¶
- Automatically coded dialogue acts distinguish male and female students' online communication
- DAC reliability varies by category: Informatives (especially negative evaluations and nonsense statements) code poorly — related
- Automatically coded dialogue acts correlate with manually coded collaborative activities as expected — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
dac-automatic-coding-agrees-with-manual-coding |
title |
Automatic DAC coding of dialogue acts agrees substantially with manual coding (79.0% agreement, kappa 0.75) |
description |
Automatic DAC coding of dialogue acts agrees substantially with manual coding (79.0% agreement, kappa 0.75) |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2008 |
recheck_by |
2011 |
generated |
by process:wiki-ingest on 2026-10-18 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — erkens-2008 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.