Students in a computationally intensive introductory physics course show large-magnitude growth in Data Practices and moderate growth in Computational Problem-Solving and Systems Thinking on written CT assessments¶

Claim · All claims
Evidence · 1 study · causal r1 · q2 · i3 large
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i3 Human-coded pre/post comparisons show highly significant growth in Data Practices (d = 1.03), Computational Problem-Solving (d = 0.52), and Systems Thinking (d = 0.45). → Sean Savage 2026

Evidence¶

Sean Savage 2026¶

Sean Savage, Anand Shanker, Grace Michlitsch, and N. Sanjay Rebello. (2026). Using LLMs to Detect Growth in Computational Thinking in Introductory Physics. arXiv preprint. https://arxiv.org/abs/2608.06200 doi:10.48550/arxiv.2608.06200

q2 · i3 · causal · r1

Paired t-test on the human-graded sample of N=50 students comparing pre- and post-instruction written responses in a calculus-based engineering physics course. The article reports "large-magnitude growth in Data Practices" with d = 1.03, and moderate growth in CPP and STP; all p-values fell below the Bonferroni-corrected threshold of 0.01.

"Students demonstrated highly significant, large-magnitude growth in Data Practices (p <0.001,d= 1.03), as well as moder- ate growth in Computational Problem-Solving (p <0.001, d= 0.52) and Systems Thinking (p <0.001,d= 0.45)."

Discussion¶

Related Claims¶

• Growth is compartmentalized by question: on the code-tracing prompt students grew in procedural code tracing while physics correctness improved only slightly — related
• A multimodal LLM achieves moderate-to-substantial agreement with human consensus on well-defined CT practices but lower agreement on multi-component constructs like Physics Correctness and Systems Thinking — related
• Systems Thinking Practices is the hardest CT construct to code reliably for both human raters and the LLM, requiring more explicitly operationalized rubrics — related
• Applied to the full dataset of 936 students, the LLM reproduced the human-observed growth trends, detecting significant growth on Q25 and Q26 but none on Q27 — related
• Modeling and Simulation Practices showed no measurable growth, attributed to a ceiling effect from students' high prior knowledge — related