Growth is compartmentalized by question: on the code-tracing prompt students grew in procedural code tracing while physics correctness improved only slightly¶

Claim · All claims
Evidence · 1 study · causal r1 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? On Q26, students showed highly significant growth in CPP (procedural code tracing) while physics correctness improved only slightly, consistent with students treating procedural coding as a syntax exercise. → Sean Savage 2026

Evidence¶

Sean Savage 2026¶

Sean Savage, Anand Shanker, Grace Michlitsch, and N. Sanjay Rebello. (2026). Using LLMs to Detect Growth in Computational Thinking in Introductory Physics. arXiv preprint. https://arxiv.org/abs/2608.06200 doi:10.48550/arxiv.2608.06200

q2 · i? · causal · r1

Question-level analysis of the human-graded sample (Table II) revealed what the authors call "a clear compartmentalization of skills": on Q25 DP and correctness grew in tandem, while on Q26 CPP growth outpaced physics correctness growth.

"Alternatively, on Q26, students showed highly significant growth in CPP (specifically procedural code trac- ing), while physics correctness only improved slightly."

Discussion¶

Related Claims¶

• Applied to the full dataset of 936 students, the LLM reproduced the human-observed growth trends, detecting significant growth on Q25 and Q26 but none on Q27 — related
• Students in a computationally intensive introductory physics course show large-magnitude growth in Data Practices and moderate growth in Computational Problem-Solving and Systems Thinking on written CT assessments — related
• A construction-focused correctness rubric underperformed behavior-focused and undefined prompts for LLM scoring of code explanations — related
• A multimodal LLM achieves moderate-to-substantial agreement with human consensus on well-defined CT practices but lower agreement on multi-component constructs like Physics Correctness and Systems Thinking — related