A reflective lesson on LLM functioning and educational impacts shifted physics students toward greater skepticism about LLM learning impacts¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? After the lesson, agreement that LLM help can leave a false sense of confidence rose from 58% to 88%, and agreement that solution generation reduces conceptual understanding rose from 63% to 85%. → Jake O'Brien 2026 q2 i? Agreement that LLMs are better problem solvers than the average physics student fell from 54% to 32%, even though instructors explicitly noted LLMs' high accuracy. → Jake O'Brien 2026 (2)

Evidence¶

Jake O'Brien 2026¶

Jake O'Brien, Margaret Ellis, Alma Robinson, John Simonetti, and Naren Ramakrishnan. (2026). Skepticism vs. Convenience: Physics Students' Perceptions and Use of Large Language Models Before and After Instruction. https://arxiv.org/abs/2609.21037 doi:10.48550/arxiv.2609.21037

q2 · i? · causal · r1

Pre/post comparison using Mann-Whitney U tests on unmatched survey responses (53 pre, 42 post). The article reports these shifts as statistically significant in Table II but prints no effect size within this passage; the shifts are treated as hypothesis-gathering rather than confirmatory.

"We found increased agreement in usage-reduced-conceptual (from 63% to 85%), usage-dependence (from 81% to 88%), and usage-false-confidence (from 58% to 88%)."

Jake O'Brien 2026 (2)¶

Jake O'Brien, Margaret Ellis, Alma Robinson, John Simonetti, and Naren Ramakrishnan. (2026). Skepticism vs. Convenience: Physics Students' Perceptions and Use of Large Language Models Before and After Instruction. https://arxiv.org/abs/2609.21037

q2 · i? · causal · r1

Post-survey Likert results for capability statements. Agreement that LLMs outperform the average physics student dropped from "54% agreement before the lesson and 32% agreement after the lesson"; the other two capability statements showed no significant shift. No effect size is printed in this passage.

"we only observe a shift toward disagreement in accuracy-above-students, which had 54% agreement before the lesson and 32% agreement after the lesson."

Discussion¶

Related Claims¶

• Despite skepticism about learning benefits, physics students report using LLMs regularly, motivated primarily by convenience and deadline pressure — related
• First-year physics students overwhelmingly believe LLMs can solve introductory physics problems with high accuracy before instruction — related
• Before instruction, physics students show skepticism about LLM learning impacts, agreeing that over-reliance risks dependence, false confidence, and reduced conceptual understanding — possibly the same claim (merge candidate)
• Most students report using LLMs to get solutions more often than they believe is ideal for learning, and this gap widened after the lesson — related