Cohen's kappa exhibits a paradox in which high observed agreement yields low kappa, making it unsuitable for pass/fail decision consistency¶

Claim · All claims
Evidence · 1 study · design r2 · q1
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly
Evidence quality · No study behind this claim is a peer-reviewed article with a DOI: it rests on a source with no registered publication record (a web page, book or unpublished report). It is kept because 2 design pages cite it; treat it as a lead to check, not an established finding.

Subclaims¶

q1 i? In a worked example with 95% observed agreement, Cohen's kappa yields only 0.64, illustrating why kappa is not commonly used to calculate DC on high stakes pass/fail exams. → Wolkowitz 2024

Evidence¶

Wolkowitz 2024¶

Wolkowitz, A. A., & Smith, R. (2024). A Practical Comparison of Decision Consistency Estimates. Practical Assessment, Research & Evaluation, 29(6). https://pareonline.net/getvn.asp?v=29&n=6

q1 · i? · design · r2

Illustrative hypothetical example of 100 examinees with 95% observed agreement and 86% expected agreement. The article attributes the paradox to the use of marginal sums, citing Feinstein & Cicchetti (1990), Cicchetti & Feinstein (1990), and Gwet (2002).

"Cohen’s κ yields a consistency statistic of 0.64, i.e., much lower than one may expected given the known agreement. This paradox is a reason that Cohen’s kappa is not commonly used (if at all) to calculate DC on a high stakes pass/fail exam"

Discussion¶

Related Claims¶

• Criterion-referenced tests require empirically-based construct validation studies — the general claim this finding bears on, with the studies behind it
• A support vector machine classifier outperformed decision tree and random forest models in predicting cognitive engagement levels of online discussion posts — related
• The LoU Interview shows high interrater reliability, and the rating procedure was streamlined over two years without loss of agreement — related
• DC estimates tend to be lowest when the cut score is set at the peak of the score distribution, where inconsistent decisions are most likely — related
• The review's evidence base comprises 42 peer-reviewed publications selected from 922 records with high inter-rater agreement (Cohen's kappa = 0.88) — related
• The interview coding achieved high inter-rater reliability, with Cohen's Kappa of 0.82 between independent coders — related
• Kappa cannot reach 1.000 when coders' marginal distributions differ, being bounded by the margins even at maximal agreement — related
• Cohen's kappa exceeds Scott's pi when coders show unequal marginal category preferences, counting those disagreements as agreement — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• Prior non-LLM automated coding approaches yielded variable and often lower agreement with human coders on teacher knowledge measures, especially PCK — related