Agent-refined frameworks improved cross-model agreement but remained below the human-refined framework and varied across foundation models¶

Claim · All claims
Evidence · 1 study · design r3 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Agreement between the two LLM agents reached QWK = 0.841 during refinement, but zero-shot agreement under their separately refined frameworks ranged from 0.655 to 0.790, below the human-refined range of about 0.85–0.89. → Lan Anh Do 2026

Evidence¶

Lan Anh Do 2026¶

Lan Anh Do, Hanling Jiang, Shuchin Aeron, Ayanna K. Thomas. (2026). Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents. https://arxiv.org/abs/2607.28651 doi:10.48550/arxiv.2607.28651

q2 · i? · design · r3

Agent–agent reliability analysis: the GPT-4o and GPT-5.2 agents reached QWK = 0.841 during refinement, while cross-model agreement under agent-refined criteria "ranged from QWK = 0.655 to 0.790", wider than under human-developed criteria (QWK ≈ 0.85–0.89).

"After applying their separately refined frameworks under zero-shot prompting, agreement ranged from QWK = 0.655 to 0.790, suggesting variability in annotation consistency across foundation models."

Discussion¶

Related Claims¶

• Trained human annotators reach near-perfect agreement on the 7-point engagement scale, improving across refinement stages — related
• Machine–machine agreement exceeds human–machine agreement but remains below human–human agreement — related
• The human-refined framework improved human agreement substantially but produced only modest gains for ICL-based LLM annotation — related
• In-context learning did not consistently improve LLM annotation reliability over zero-shot prompting for engagement coding — related