First-year physics students overwhelmingly believe LLMs can solve introductory physics problems with high accuracy before instruction¶

Claim · All claims
Evidence · 1 study · associational r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2031: baselines and conditions change

Subclaims¶

q2 i? Before instruction, 79% of surveyed physics students agreed that LLMs can solve introductory physics problems with high accuracy. → Jake O'Brien 2026

Evidence¶

Jake O'Brien 2026¶

Jake O'Brien, Margaret Ellis, Alma Robinson, John Simonetti, and Naren Ramakrishnan. (2026). Skepticism vs. Convenience: Physics Students' Perceptions and Use of Large Language Models Before and After Instruction. https://arxiv.org/abs/2609.21037 doi:10.48550/arxiv.2609.21037

q2 · i? · associational · r1

Pre-lesson Likert survey of 52 first-year physics majors in a required FYE course, reported as agreement/disagreement percentages. The pre-survey found "79% agreement and 13% disagreement" that LLMs solve introductory physics problems with high accuracy; no effect size applies to this descriptive result.

"The vast majority of students believe LLMs to be capable of solving introductory physics problems with high accuracy (accuracy-intro-physics), with 79% agreement and 13% disagreement."

Discussion¶

Related Claims¶

• Most students report using LLMs to get solutions more often than they believe is ideal for learning, and this gap widened after the lesson — related
• Before instruction, physics students show skepticism about LLM learning impacts, agreeing that over-reliance risks dependence, false confidence, and reduced conceptual understanding — related
• A reflective lesson on LLM functioning and educational impacts shifted physics students toward greater skepticism about LLM learning impacts — related
• A general-purpose LLM used as a tutor without pedagogical prompting overwhelmingly provides high assistance, with more than 95% of responses classified as Explaining or Solving — related