Future directions: fine-tuning on human-refined frameworks and hybrid human–LLM or multi-agent annotation systems¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The authors recommend two directions to improve scalability and annotation support: first, examine whether LLMs can be fine-tuned using the human-refined framework and annotated data to assess whether models acquire implicit knowledge not fully captured by explicit criteria; second, explore "hybrid systems in which human coders and LLM agents collaborate, and multi-agent systems in which multiple agents co-ordinate without human involvement". Both may refine frameworks, identify challenging cases, and support quality control while reducing human annotator workload.

Design Implications¶

Context¶

Requirements¶

• Human-refined framework and annotated data for fine-tuning; coordination mechanisms for multi-agent systems

Constraints¶

• A key open question is whether LLMs can make reliable autonomous decisions and align with human judgments

Target Learners¶

• researchers annotating collaborative learning discourse

Target Learning Goals¶

• reliable, scalable annotation of cognitive engagement

Related Strategies¶

• Incorporate an expert human in the loop to support LLM coding of difficult adjacent coding categories

Examples¶

-

Key Sources¶

• Lan Anh Do, Hanling Jiang, Shuchin Aeron, Ayanna K. Thomas. (2026). Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents. https://arxiv.org/abs/2607.28651