GPT-5 used as a third coder introduced a calculation_check category absent from the human codebook, covering 28% of messages and exposing an unnamed recurring request type¶

Claim · All claims
Evidence · 1 study · qualitative r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? GPT-5 introduced calculation_check for messages in which students submitted completed work and asked about computational accuracy, a category the human coders had distributed across two improvement-request labels. → Zahra 2025

Evidence¶

Zahra 2025¶

Zahra, F. T. (2025). Three Pathways of Student-AI Interaction: Constraint-First Design for Higher-Order Thinking. Manuscript under blind review; framework first presented at EAAI-25, Philadelphia, PA. https://arxiv.org/abs/2610.00338 doi:10.48550/arxiv.2610.00338

q2 · i? · qualitative · r2

Supplementary coding exercise in which GPT-5 applied the existing codebook to the same 50 messages and introduced "calculation_check" for 28% of messages. The scheme divergence reduced comparability and produced low pairwise kappas; no effect size is printed.

"It introduced calculation_check for messages in which students submitted completed work and asked about computational accuracy. The human codebook had distributed these messages across two improvement-request categories, so the new label made visible a recurring feature that we had not named separately."

Discussion¶

Related Claims¶

• Systematic human coding added five codes the LLM-assisted phases had not surfaced, extending the codebook from 67 to 72 items and 18 to 19 categories — related
• LLM-assisted inductive qualitative coding carries risks of superficial themes, broad or redundant codes, and hallucinated interpretations, so LLMs should augment rather than replace human researchers — a broader claim this one bears on
• Human review remains essential after GPT-based codebook generation, producing a final refined SRL codebook of eleven constructs — related
• Fully inductive LLM codebook development risks importing unexamined sensitizing concepts, such as folk theories and scientific misconceptions, from the model's training data — related
• Human-LLM agreement on a complex multi-label codebook falls well below human-human agreement, while LLM-LLM agreement is comparable to human-human agreement — related