During member checks, multilingual 8th-grade students contested both LLM classifications of their talk and the coding scheme itself, especially the Off-task and Claim codes¶

Claim · All claims
Evidence · 1 study · qualitative r3 · q2
Currency · no dated study recorded · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Student member checks revealed misalignments between how students understood their own talk and how the LLM and coding scheme classified it, with students contesting Off-task and Claim labels by foregrounding their intentions. → Liliana Santos-Deonizio 2027

Evidence¶

Liliana Santos-Deonizio 2027¶

Liliana Santos-Deonizio, James Malamut, Ramón Antonio Martínez, and Dorottya Demszky. (2027). When Youth Enter The Chat: An Epistemic Shift in the Validation of LLM-Based Measures of Student Talk. In Proceedings of TBD (CSCW '27). ACM, New York, NY, USA. https://arxiv.org/abs/2608.23780 doi:10.48550/arxiv.2608.23780

q2 · i? · qualitative · r3

Case-study member checks with four focal students in one 8th-grade math classroom, tabulating disagreements with LLM classifications (Table 4). Students were "especially vocal about the Claim and Off-task codes" and pushed back by foregrounding intentions behind their utterances.

"While students generally agreed with the LLM classifications, the moments of contestation—and the reasons they gave for them—highlighted misalignments with the LLMs’ classification of a given utterance, and with the coding scheme itself."

Discussion¶

Related Claims¶

• The coding scheme lacked codes for guessing and brainstorming, so students' exploratory solution talk was misclassified as Claim — a narrower finding that bears on this claim
• Ximena contested three Off-task labels on utterances that referenced math, arguing her math-related frustration and self-talk were on task — a narrower finding that bears on this claim
• Ethnographically-oriented methods surface five dimensions of classroom context that text-only transcripts omit, including multimodal communication and relationships — related
• LLM-based transcript classifications agreed highly with human judgment for explicit questions (κ = .778) and math talk (κ = .722), but only moderately for need for help (κ = .562) and confusion (κ = .531) — related
• Agreement metrics and expert quality judgments diverge in both directions: human consensus can encode shared conservative bias that the verifier rejects in favor of LLM coding — related
• LLM-assisted inductive qualitative coding carries risks of superficial themes, broad or redundant codes, and hallucinated interpretations, so LLMs should augment rather than replace human researchers — a broader claim this one bears on
• Model and prompt choice account for only a small share of misalignment error, which concentrates in transcript-conditioned higher-order interactions — related