All automated models, including GradeOpt, were most challenged by knowledge of teaching mathematics items (GradeOpt κ = .59) and least challenged by knowledge of students' mathematical thinking (κ = .79)¶
Claim · All claims
Evidence · 1 study · designr3·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? GradeOpt's Cohen's κ varied across PCK subdomains: .66 for knowledge of mathematical tasks, .79 for knowledge of students' mathematical thinking, and .59 for knowledge of teaching mathematics. → Copur-Gencturk 2026
Evidence¶
Copur-Gencturk 2026¶
Copur-Gencturk, Y., Moreno, K., Chu, Y., Li, H., & Tang, J. (2026). Automated coding of content and pedagogical content knowledge of mathematics using a multi-agent large language model. ZDM – Mathematics Education. https://doi.org/10.1007/s11858-026-01796-2
q2 · i? · design · r3
Subdomain-level results from the same test-set evaluation of nine PCK items. The article attributes the difficulty to complex teaching scenarios and cognitive demands such as applying a teaching strategy and explaining the rationale, noting these items were also challenging for human coders.
"GradeOpt's performance was acceptable for overall PCK (Cohen's κ = .68) but varied across PCK subdomains, with Cohen's κ values of .66 for knowledge of mathematical tasks, .79 for knowledge of students' mathematical thinking, and .59 for knowledge of teaching mathematics. These results indicate that all models, including GradeOpt, were particularly challenged by coding the knowledge of teaching mathematics items."
Discussion¶
Related Claims¶
- The multi-agent LLM GradeOpt achieved substantial agreement with human coders on overall PCK items (κ = .68, QWK = .79) and outperformed RoBERTa, SBERT, and naïve GPT-4o prompting — related
- Gaps between Cohen's κ and QWK indicate GradeOpt often assigned codes adjacent to human codes rather than entirely discrepant ones — related
- Prior non-LLM automated coding approaches yielded variable and often lower agreement with human coders on teacher knowledge measures, especially PCK — related
- On CK items, GradeOpt showed near-perfect alignment with human codes (κ = .88 and .85; QWK = .89 and .94), comparable to human–human agreement — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
pck-subdomain-variation-ktm-hardest |
title |
All automated models, including GradeOpt, were most challenged by knowledge of teaching mathematics items (GradeOpt κ = .59) and least challenged by knowledge of students' mathematical thinking (κ = .79) |
description |
All automated models, including GradeOpt, were most challenged by knowledge of teaching mathematics items (GradeOpt κ = .59) and least challenged by knowledge of students' mathematical thinking (κ = .79) |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — copur-gencturk-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.