Human-human agreement (average κ = 0.644) was lower than the best LLM-LLM agreement (average κ = 0.856) across constructs¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Human-human agreement across constructs (average κ = 0.644) was lower than the best LLM-LLM agreement (average κ = 0.856), partially supporting H3a. → Ober 2026
Evidence¶
Ober 2026¶
Ober, T. M., Zhang, S., Zapata-Rivera, D., Schroeder, N. L., & Botelho, A. F. (2026). Using LLMs to Identify Indicators of Persistence from Students' Dialogues with a Pedagogical Agent. Journal of Educational Data Mining, Volume 18, No 1. https://doi.org/10.17605/osf.io/s85ck
q2 · i? · design · r2
Comparison of human-human inter-rater reliability against the best LLM-LLM agreement across the five constructs. Human coders showed strong agreement for self-efficacy (κ = 0.886) but moderate agreement for interest (κ = 0.675) and persistence (κ = 0.514).
"Yet at the same time, human-human agreement (average κ = 0.644) was actually lower than the best LLM-LLM agreement (average κ = 0.856) across constructs. (H3a partially supported)"
Discussion¶
Related Claims¶
- Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related
- Constructs with higher operational clarity show higher overall coder agreement, and low clarity harms human coder agreement more than LLM agreement — related
- Same-model LLM configuration pairs agree more than cross-model pairs, and agreement decreases monotonically as temperature difference increases — related
- In blind verification, model selection matters more than source type: Bradley-Terry ranking interleaves human coders and LLMs, with per-model win rates spanning significant human advantage to numerical LLM advantage — related
- Cross-model agreement on assertions is moderate (median 0.401) and higher among construct-derived than corpus-derived assertions — related
- LLM-human factor-structure similarity is consistently lower than human-human similarity on both assessments — related
- Across 468 data points, human coders, LLMs, and both standards agreed on the same value in 280 cases (59.8%) — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
human-human-agreement-lower-than-llm-llm |
title |
Human-human agreement (average κ = 0.644) was lower than the best LLM-LLM agreement (average κ = 0.856) across constructs |
description |
Human-human agreement (average κ = 0.644) was lower than the best LLM-LLM agreement (average κ = 0.856) across constructs |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-08 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — ober-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.