Kappa agreement with human experts shows a system is neither much worse nor much better than experts, but not that experts are a desirable comparison basis¶
Claim · All claims
Evidence · 1 study · reviewr2·q1
Currency · newest study 2015 · not yet checked against its sources · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q1 i? Kappa measures of agreement between an adaptive system and human experts indicate the system behaves neither much worse nor much better than experts, but do not establish the experts as a desirable basis of comparison. → Jim Greer & Mary Mark 2015
Evidence¶
Jim Greer & Mary Mark 2015¶
Jim Greer & Mary Mark. (2015). Evaluation Methods for Intelligent Tutoring Systems Revisited. Int J Artif Intell Educ 26:387–392. https://doi.org/10.1007/s40593-015-0043-2
q1 · i? · review · r2
The authors' methodological argument (type e) about kappa-based criterion evaluation: kappa is "a consensus estimate of inter-rater reliability" rarely used for tests of statistical significance, and low expert agreement gives wide latitude for system decisions.
"When comparing a system with a human expert, a good Kappa measure tells us that the system is behaving neither much worse nor much better than a human expert, but not whether the human experts are a desirable basis of comparison."
Discussion¶
Related Claims¶
- LLM agreement with human examiners is limited in fast mode and improves in reasoning or thinking modes — related
- Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
- Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
- A Gemini-based creativity autorater scores real students' complex multimedia creativity tasks on par with human experts (item Kappa 0.66; total-score Pearson r = 0.88) — related
- Confidence-aware selective test-time scoring achieves the best average agreement with expert rubric scoring across six NGSS drawing items — related
- The rule-based recipe annotation system reaches near-perfect inter-annotator agreement (Cohen's Kappa 0.85) and average accuracy 0.95 in a validation on sampled recipe segments. — related
- Human Graders of Open-Ended Work Agree With Each Other Less Than Treating One Grade as Ground Truth Implies — related
- GAME's marks agree with a human marker's marks on student programming exercises and assignments — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
kappa-expert-agreement-limits |
title |
Kappa agreement with human experts shows a system is neither much worse nor much better than experts, but not that experts are a desirable comparison basis |
description |
Kappa agreement with human experts shows a system is neither much worse nor much better than experts, but not that experts are a desirable comparison basis |
status |
review |
evidence_strength |
weak |
newest_evidence |
2015 |
generated |
by process:wiki-ingest on 2026-10-28 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — jim-greer-mary-mark-2015 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.