The human-refined framework improved human agreement substantially but produced only modest gains for ICL-based LLM annotation¶
Claim · All claims
Evidence · 1 study · designr3·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Moving from Criteria Version 1 to Version 2 raised human–human agreement (Δκ = 0.10) but changed human–machine QWK only by about 0.01–0.04. → Lan Anh Do 2026
Evidence¶
Lan Anh Do 2026¶
Lan Anh Do, Hanling Jiang, Shuchin Aeron, Ayanna K. Thomas. (2026). Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents. https://arxiv.org/abs/2607.28651 doi:10.48550/arxiv.2607.28651
q2 · i? · design · r3
Comparative reliability analysis across annotation stages: GPT-5.2 with ICL prompting showed human–machine QWK of 0.593–0.655 across stages and criteria versions, with the human-refined framework yielding "only small changes in reliability" for LLMs.
"The human-refined framework produced only small changes in reliability from Criteria Version 1 to Version 2 (overall ∆QWK≈0.01–0.04), in contrast to the larger gains observed for human–human agreement."
Discussion¶
Related Claims¶
- Agent-refined frameworks improved cross-model agreement but remained below the human-refined framework and varied across foundation models — related
- Trained human annotators reach near-perfect agreement on the 7-point engagement scale, improving across refinement stages — related
- Machine–machine agreement exceeds human–machine agreement but remains below human–human agreement — related
- In-context learning did not consistently improve LLM annotation reliability over zero-shot prompting for engagement coding — related
- LLM ratings of classroom transcripts are more correlated with each other than with expert human ratings, across the same and different tasks — related
- Developmental representations remain relatively stable across translated bilingual narratives — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
human-refined-framework-gains-humans-not-llms |
title |
The human-refined framework improved human agreement substantially but produced only modest gains for ICL-based LLM annotation |
description |
The human-refined framework improved human agreement substantially but produced only modest gains for ICL-based LLM annotation |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — lan-anh-do-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.