Applied to the full dataset of 936 students, the LLM reproduced the human-observed growth trends, detecting significant growth on Q25 and Q26 but none on Q27¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across over 2,800 responses from N = 936 students, the LLM detected highly statistically significant growth (p < 0.001) on Q25 and Q26, while Q27 showed no measurable growth on any parameter. → Sean Savage 2026

Evidence¶

Sean Savage 2026¶

Sean Savage, Anand Shanker, Grace Michlitsch, and N. Sanjay Rebello. (2026). Using LLMs to Detect Growth in Computational Thinking in Introductory Physics. arXiv preprint. https://arxiv.org/abs/2608.06200 doi:10.48550/arxiv.2608.06200

q2 · i? · design · r2

The validated multimodal LLM pipeline was deployed across the full dataset of 936 students, evaluating over 2,800 total responses with majority voting across three runs. The article reports significant growth on Q25 and Q26 and no measurable growth on Q27, including STP and Physics Correctness.

"When applied to the full dataset ofN= 936students, the LLM analysis confirmed the macroscopic trends observed in the human baseline. By evaluating over 2,800 total re- sponses, the model detected highly statistically significant growth (p <0.001) across Q25 and Q26."

Discussion¶

Related Claims¶

• Growth is compartmentalized by question: on the code-tracing prompt students grew in procedural code tracing while physics correctness improved only slightly — related
• Students in a computationally intensive introductory physics course show large-magnitude growth in Data Practices and moderate growth in Computational Problem-Solving and Systems Thinking on written CT assessments — related
• Modeling and Simulation Practices showed no measurable growth, attributed to a ceiling effect from students' high prior knowledge — related