Human and agent framework revisions concentrate on the mid-level engagement categories (Levels 3–5), indicating greater ambiguity there than at scale endpoints¶

Claim · All claims
Evidence · 1 study · design r3 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Human disagreements occurred most often between Levels 4 and 5 (24% of disagreement pairs), and both humans and agents made the most revisions to Levels 3–5. → Lan Anh Do 2026

Evidence¶

Lan Anh Do 2026¶

Lan Anh Do, Hanling Jiang, Shuchin Aeron, Ayanna K. Thomas. (2026). Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents. https://arxiv.org/abs/2607.28651 doi:10.48550/arxiv.2607.28651

q2 · i? · design · r3

Disagreement analysis of human confusion matrices plus agent refinement counts (Figure 3): bidirectional confusion between adjacent mid-levels (e.g., 11–23% of Level 4 items coded Level 5 by the other annotator); agents likewise modified Levels 3–5 most, mirroring human refinement.

"Human disagreements occurred most frequently between Levels 4 and 5 (24% of all disagreement pairs), followed by Levels 5 and 6 (17%) and Levels 3 and 4 (16%), although this pattern may partly reflect the higher frequency of Level 5 behaviors."

Discussion¶

Related Claims¶

• The ICAP Framework was found effective for assessing potential for active learning in an asynchronous online environment and identifying where design improvements are needed — the general claim this finding bears on, with the studies behind it
• Feedback givers agree on exact urgency tiers only about a fifth of the time, with most disagreements off by a single tier — related
• Students who experienced annotation subjectivity firsthand most frequently requested ways to reduce disagreement, indicating they had not internalised disagreement as meaningful signal — related
• Round II raised exact question-part agreement from about 63% to about 70% and cut parts differing by more than one point from about 13% to 7%, while exact partial-credit scoring remained hardest — related
• Focused rubric refinement that made required physics and scoring conditions explicit reduced AI–official disagreements on targeted questions (e.g., Grover item MAD 1.1 to 0.4 marks, r 0.70 to 0.82) — related
• CS/SE educators rate ANVIL analogies highly and animations as generally faithful, with disagreement concentrated in borderline and visual-clarity judgments — related
• The revised Round II grading workflow reduced AI over-awarding and total-score MAD, most clearly for OE1 (7.1% to 4.8%) and QM (9.4% to 3.8%) — related
• Student utterances requiring instructional scaffolding rose from about 16.1% early in the consultation to about 34.8% late — related