Cosine similarity has opposing effects on LLM coding accuracy: lower similarity within codebooks helps, higher similarity within excerpts helps¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2 · i1 small
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i1 Higher cosine similarity within the codebook negatively affects initial coding accuracy (effect size -0.276, p<0.0001). → Jeongyeon Kim and John Mitchell 2026 q2 i1 Higher cosine similarity within the excerpt positively affects initial coding accuracy (effect size 0.189, p<0.0001). → Jeongyeon Kim and John Mitchell 2026 (2)

Evidence¶

Jeongyeon Kim and John Mitchell 2026¶

Jeongyeon Kim and John Mitchell. (2026). How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding. arXiv. https://arxiv.org/abs/2609.11109 doi:10.48550/arxiv.2609.11109

q2 · i1 · design · r2

Mixed-effects analysis of dataset characteristics across the four datasets. The codebook-similarity row reports "(-11.738, -0.276)", p<0.0001 with effect size -0.276; the article interprets that varied codebook language aids contextual understanding of the code.

"Cosine Similarity of Codebook < 0.0001 **** (-11.738, -0.276) The less similar the semantics of words in each codebook, the higher the initial accuracy of automated coding."

Jeongyeon Kim and John Mitchell 2026 (2)¶

Jeongyeon Kim and John Mitchell. (2026). How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding. arXiv. https://arxiv.org/abs/2609.11109

q2 · i1 · design · r2

Same mixed-effects analysis. The excerpt-similarity row reports "(7.372, 0.189)", p<0.0001 with effect size 0.189; the article concludes "coding accuracy improves when codebooks employ diverse language while the target data remains consistent."

"Cosine Similarity of Excerpt < 0.0001 **** (7.372, 0.189) The more similar the semantics of words in excerpt to code, the higher the initial accuracy of automated coding."

Discussion¶

Related Claims¶

• Shorter codebooks and shorter data excerpts yield higher initial LLM coding accuracy, and a smaller specialization gap between codebook and excerpt helps — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• LLM-assisted inductive qualitative coding carries risks of superficial themes, broad or redundant codes, and hallucinated interpretations, so LLMs should augment rather than replace human researchers — related