Modeling and Simulation Practices showed no measurable growth, attributed to a ceiling effect from students' high prior knowledge¶

Claim · All claims
Evidence · 1 study · causal r1 · q2 · i0 negligible
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i0 Modeling and Simulation Practices showed no statistical growth (p = 1.000, d = 0.00), which the authors attribute to an assessment ceiling effect. → Sean Savage 2026

Evidence¶

Sean Savage 2026¶

Sean Savage, Anand Shanker, Grace Michlitsch, and N. Sanjay Rebello. (2026). Using LLMs to Detect Growth in Computational Thinking in Introductory Physics. arXiv preprint. https://arxiv.org/abs/2608.06200 doi:10.48550/arxiv.2608.06200

q2 · i0 · causal · r1

Paired t-test on the human-graded sample (N=50) found no growth in MSP, with a printed effect size of d = 0.00 and p = 1.000. The pre-instruction mean for MSP on Q27 was 1.62 against a rubric maximum of 2, which the authors call "an assessment ceiling effect".

"However, Modeling and Simulation Practices showed no statistical growth (p= 1.000,d= 0.00). This lack of change reflects an assessment ceiling effect, in which students entered the course already able to list simulation pa- rameters, leaving no room for measurable growth on this spe- cific prompt."

Discussion¶

Related Claims¶

• Students in a computationally intensive introductory physics course show large-magnitude growth in Data Practices and moderate growth in Computational Problem-Solving and Systems Thinking on written CT assessments — related
• Applied to the full dataset of 936 students, the LLM reproduced the human-observed growth trends, detecting significant growth on Q25 and Q26 but none on Q27 — related
• Correlations between estimated and true scores diminish over time mainly with easy items, especially with a true slope of .5 and only four items, likely due to ceiling effects — related