Prior non-LLM automated coding approaches yielded variable and often lower agreement with human coders on teacher knowledge measures, especially PCK¶
Claim · All claims
Evidence · 1 study · designr3·q1
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q1 i? The review reports that Kersting et al. (2014) achieved quadratic weighted kappa values of .56 to .64 for mathematical content but .36 to .47 for student thinking and suggestions for improvement, and Wahlen et al. (2020) found automated–human agreement of .38 to .77 versus human–human agreement of .80 to .97. → Copur-Gencturk 2026
Evidence¶
Copur-Gencturk 2026¶
Copur-Gencturk, Y., Moreno, K., Chu, Y., Li, H., & Tang, J. (2026). Automated coding of content and pedagogical content knowledge of mathematics using a multi-agent large language model. ZDM – Mathematics Education. https://doi.org/10.1007/s11858-026-01796-2
q1 · i? · design · r3
Narrative review synthesis in the literature review, citing Wahlen et al. (2020) on economics-education PCK items, where human–human quadratic weighted kappa ranged from .80 to .97. The review reports the printed kappa ranges; the reviewed studies' designs are not described here beyond what the article states.
"In contrast, agreement between au- tomated coding and human coders was consistently lower and more variable across items, with quadratic weighted kappa values ranging from .38 to .77."
Discussion¶
Related Claims¶
- The multi-agent LLM GradeOpt achieved substantial agreement with human coders on overall PCK items (κ = .68, QWK = .79) and outperformed RoBERTa, SBERT, and naïve GPT-4o prompting — related
- LLM agreement with human examiners is limited in fast mode and improves in reasoning or thinking modes — related
- Comparison models (RoBERTa, SBERT, naïve GPT-4o prompting) failed to reach acceptable agreement on PCK items, with Cohen's κ at or below .55 — related
- All automated models, including GradeOpt, were most challenged by knowledge of teaching mathematics items (GradeOpt κ = .59) and least challenged by knowledge of students' mathematical thinking (κ = .79) — related
- On CK items, GradeOpt showed near-perfect alignment with human codes (κ = .88 and .85; QWK = .89 and .94), comparable to human–human agreement — related
- Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related
- Perceptions of feedback characteristics vary between researcher and teacher coders, with agreement shaped by professional background — related
- Cohen's kappa exhibits a paradox in which high observed agreement yields low kappa, making it unsuitable for pass/fail decision consistency — related
- Studies of teachers' SRL knowledge use heterogeneous, often implicit definitions, limiting comparability — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
prior-automated-coding-variable-lower-agreement-pck |
title |
Prior non-LLM automated coding approaches yielded variable and often lower agreement with human coders on teacher knowledge measures, especially PCK |
description |
Prior non-LLM automated coding approaches yielded variable and often lower agreement with human coders on teacher knowledge measures, especially PCK |
status |
review |
evidence_strength |
weak |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — copur-gencturk-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.