The emergent LLM coding pipeline reaches substantial agreement with expert human coding (Cohen's κ = 0.78, raw agreement 84%) on a stratified 10% sample¶

Claim · All claims
Evidence · 1 study · qualitative r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? LLM-generated discourse labels validated against an expert who independently coded a stratified random sample of 287 student messages (10% of the corpus) achieved Cohen's κ = 0.78 (raw agreement 84%), within the range conventionally interpreted as substantial agreement. → Syed Furqan Abbas Hashmi and N. Sanjay Rebello 2026

Evidence¶

Syed Furqan Abbas Hashmi and N. Sanjay Rebello 2026¶

Syed Furqan Abbas Hashmi and N. Sanjay Rebello. (2026). A bottom-up taxonomy of student discourse with a Socratic AI physics tutor. https://arxiv.org/abs/2608.07373 doi:10.48550/arxiv.2608.07373

q2 · i? · qualitative · r2

Reliability validation of the LLM coding pipeline: an expert independently coded a stratified random sample of 287 student messages (10% of the corpus) under the same prompt instructions, and labels were mapped onto the consolidated category set. Disagreements clustered on borderline moves such as confirmation versus restatement.

"agreement was Cohen's κ= 0.78(raw agreement 84%), which falls within the range conventionally interpreted as substantial agreement and is consistent with reliability levels reported for human and ma- chine coding in recent PER work"

Discussion¶

Related Claims¶

• Student discourse with a Socratic AI physics tutor concentrates sharply: of 357 consolidated categories, the top 25 cover roughly half of all student turns — related
• The LLM-judge scoring pipeline for the two metrics aligns with human labels and human raters — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related
• An independent human second coder reached moderate agreement (kappa = 0.395) with the AI-assisted primary coding on the five-category outcome — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• AI assessors show emerging reliability: acceptable AI-human rater agreement (Cohen's 0.78) in a 12-educator case study and human-comparable feedback helpfulness ratings with 45 learners — related
• The qualitative coding achieved 82% exact interrater agreement, indicating acceptable reliability — related
• Human inter-rater agreement for FRISCO essay scoring was reliable, with mean kappa 0.78–0.84 and percentage of agreement above 75% in all six dimensions — related
• A multimodal LLM achieves moderate-to-substantial agreement with human consensus on well-defined CT practices but lower agreement on multi-component constructs like Physics Correctness and Systems Thinking — related