Students engaged deeply with LLM feedback, with coded evidence of reflection, rebuttal, and acceptance in the large majority of responses¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Blind coders found evidence of reflection in 92.7% of student responses to LLM feedback. → Daniel M. Oppenheimer 2025 q2 i? Coders found evidence of rebuttal in 87.8% and of acceptance in 93.6% of responses. → Daniel M. Oppenheimer 2025 (2)

Evidence¶

Daniel M. Oppenheimer 2025¶

Daniel M. Oppenheimer, Trent N. Cash, Allison E. Connell Pensky. (2025). You've Got AI Friend in Me: LLMs as Collaborative Learning Partners. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-025-00521-6

q2 · i? · design · r2

Two blind undergraduate research assistants coded each student's response on three dimensions, including whether the student reflected by "evaluating the quality of the output, comparing the LLM's output to their own thoughts, or explicitly discussing the LLM's output." Inter-rater reliability was high (ks = 0.81 − 0.89).

"On average, the coders found evidence of reflection in 92.7% of responses."

Daniel M. Oppenheimer 2025 (2)¶

Daniel M. Oppenheimer, Trent N. Cash, Allison E. Connell Pensky. (2025). You've Got AI Friend in Me: LLMs as Collaborative Learning Partners. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-025-00521-6

q2 · i? · design · r2

In the same blind coding of student responses, rebuttal was coded as "arguing against any of the feedback" or "explaining why the LLM's response may not be valid/appropriate"; acceptance appeared in 93.6% of responses. The authors conclude students were "making active decisions about what feedback to accept and what feedback to rebut."

"On average, the coders found evidence of rebuttal in 87.8% of responses."

Discussion¶

Related Claims¶

• Text-based reflective use from Copilot Critical Engagement responses does not predict productive engagement, with limited coder reliability and a TA ceiling effect (exploratory null) — related
• Blind expert verification shows no overall preference for human over LLM qualitative coding when sources are judged symmetrically — related
• In blind verification, model selection matters more than source type: Bradley-Terry ranking interleaves human coders and LLMs, with per-model win rates spanning significant human advantage to numerical LLM advantage — related
• Student-AI exchanges in an undergraduate research methods course are empirically classifiable into three interaction paths, with Passive Review most frequent (46%) and Strategic Dialogue 36% — related
• Human-LLM agreement remains low across constructs (κ = 0.000 to 0.419), with self-efficacy showing the best alignment and Prior KSAs none — related
• LLM coders replicate many human discursive moves but lack questioning, adaptive responsiveness, and emotional expressiveness — related
• GPT4-coded sentiment of students' open-ended general feedback about the intervention was overwhelmingly positive — related