In-context learning did not consistently improve LLM annotation reliability over zero-shot prompting for engagement coding¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across criteria versions and models, ICL and zero-shot prompting produced similar agreement, and zero-shot sometimes slightly outperformed ICL. → Lan Anh Do 2026

Evidence¶

Lan Anh Do 2026¶

Lan Anh Do, Hanling Jiang, Shuchin Aeron, Ayanna K. Thomas. (2026). Measuring Cognitive Engagement in Collaborative Discourse with an Extended ICAP Framework: Comparing Human Annotation, In-Context Learning, and Reflective LLM Agents. https://arxiv.org/abs/2607.28651 doi:10.48550/arxiv.2607.28651

q2 · i? · causal · r2

Null comparison within the same annotation study: for GPT-5.2 and GPT-4o under human-refined criteria, ICL and zero-shot QWK values were closely matched (e.g., GPT-5.2 Cri-V1 overall 0.631 ICL vs 0.629 zero-shot), a null result with no printed effect size.

"Differences between ICL and zero-shot prompting were also modest and sometimes favored zero-shot, indicating that few-shot prompting with human examples did not consistently improve human–machine reliability on this task."

Discussion¶

Related Claims¶

• Agent-refined frameworks improved cross-model agreement but remained below the human-refined framework and varied across foundation models — related
• Curricular chain-of-thought prompting improves competency-classification accuracy over zero-shot, with gains concentrated in larger models, while definition-based prompting does not improve performance — related
• Adding regex-based rules to few-shot LLM labeling improves agreement with human annotation (0.818 to 0.852 overall) — related
• Zero-shot prompt type has minimal impact on LLM-human coding concordance — related
• Machine–machine agreement exceeds human–machine agreement but remains below human–human agreement — related
• Few-shot prompting does not improve diagnostic F1 or accuracy but reduces evidence over-attribution and false-attribution — related
• Gemini 2.0 Flash validated as a reliable LLM annotator of feedback focus types — related
• Few-shot prompting does not uniformly improve LLM performance and can degrade subskill mastery estimates, most notably for phonology — a narrower finding that bears on this claim
• The human-refined framework improved human agreement substantially but produced only modest gains for ICL-based LLM annotation — related
• Few-shot prompting modestly improves LLM-human score agreement, inconsistently across datasets and models — related