LLM use for debugging is significantly negatively correlated with student final grades¶

Claim · All claims
Evidence · 1 study · associational r1 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Use of LLMs for debugging correlates negatively with final grades (Spearman's rho = -0.360, p = 0.021). → Jošt 2024

Evidence¶

Jošt 2024¶

Jošt, G.; Taneski, V.; Karakatič, S. (2024). The Impact of Large Language Models on Programming Education and Student Learning Outcomes. Appl. Sci. 14, 4115. https://doi.org/10.3390/app14104115

q3 · i? · associational · r1

Spearman correlation analysis from the same experiment; the entirely negative bootstrap confidence interval (−0.628, −0.011) supports the outcome. The article prints "Spearman's rho = −0.360, p = 0.021".

"H2c was strongly supported by a significant inverse correlation between debugging with LLMs and final grades (Spearman's rho = −0.360, p = 0.021)"

Discussion¶

Related Claims¶

• Higher average LLM use across all tasks is significantly negatively correlated with final grades in undergraduate programming education — a broader claim this one bears on
• LLM use for code generation is significantly negatively correlated with student final grades — related
• LLM use for seeking additional explanations does not significantly impact final grades — related
• A downward trend in final grades is observed with increased average LLM use across all task types
• By the final survey wave, using LLMs for debugging and understanding concepts was nearly universal among surveyed students