Incorporate an expert human in the loop to support LLM coding of difficult adjacent coding categories¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article's discussion recommends exploring human-in-the-loop support when multi-agent refinement alone leaves adjacent coding categories confounded. It states that "Consulting a trained human rater about points of difficulty may substantially enhance the performance of the LLM," because human coders can offer deeper insight into distinctions between overall and adjacent levels than an additional LLM-driven agent. The strategy applies after automated refinement when κ–QWK gaps reveal persistent adjacent-category confusion on complex items.

Design Implications¶

Context¶

Requirements¶

• Access to a trained human rater with expertise in the coding construct
• Coding manuals and disagreement data identifying the items where the model struggles

Constraints¶

• The article frames this as a direction to explore, not a tested manipulation in this study; the items where it would apply were also challenging for human coders

Target Learners¶

• U.S. middle school mathematics teachers (whose responses are being coded)

Target Learning Goals¶

• Reliable automated coding of teachers' pedagogical content knowledge from open-ended responses

Related Strategies¶

• Future directions: fine-tuning on human-refined frameworks and hybrid human–LLM or multi-agent annotation systems
• Integrate GPT models into LA-supported feedback systems with an iterative instructor-student validation loop to improve reliability

Examples¶

-

Key Sources¶

• Copur-Gencturk, Y., Moreno, K., Chu, Y., Li, H., & Tang, J. (2026). Automated coding of content and pedagogical content knowledge of mathematics using a multi-agent large language model. ZDM – Mathematics Education. https://doi.org/10.1007/s11858-026-01796-2