Comparison models (RoBERTa, SBERT, naïve GPT-4o prompting) failed to reach acceptable agreement on PCK items, with Cohen's κ at or below .55¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? RoBERTa and SBERT achieved Cohen's κ between .00 and .39 and .00 and .55 respectively on PCK items, and naïve GPT-4o prompting ranged from .06 to .54. → Copur-Gencturk 2026

Evidence¶

Copur-Gencturk 2026¶

Copur-Gencturk, Y., Moreno, K., Chu, Y., Li, H., & Tang, J. (2026). Automated coding of content and pedagogical content knowledge of mathematics using a multi-agent large language model. ZDM – Mathematics Education. https://doi.org/10.1007/s11858-026-01796-2

q2 · i? · design · r2

Results section comparison on the same held-out test set of teachers' PCK responses. The article states RoBERTa and SBERT "did not achieve acceptable levels of performance" and the Naïve Prompt approach "also experienced difficulty coding these responses effectively."

"The two commonly used pretrained, rule-based methods, RoBERTa and SBERT, did not achieve acceptable levels of performance, particularly in terms of Cohen's κ (.00 to .39 for RoBERTa and .00 to .55 for SBERT). Among the LLM-based methods, the Naïve Prompt approach also experienced difficulty coding these responses effectively (Cohen's κ ranging from .06 to .54)."

Discussion¶

Related Claims¶

• The multi-agent LLM GradeOpt achieved substantial agreement with human coders on overall PCK items (κ = .68, QWK = .79) and outperformed RoBERTa, SBERT, and naïve GPT-4o prompting — related
• Prior non-LLM automated coding approaches yielded variable and often lower agreement with human coders on teacher knowledge measures, especially PCK — related
• Sentence embedding models reach only moderate agreement and need human supervision for open-ended grading — related