Higher average LLM use across all tasks is significantly negatively correlated with final grades in undergraduate programming education¶

Claim · All claims
Evidence · 1 study · associational r1 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Average LLM use correlates negatively with final grades (Spearman's rho = -0.347, p = 0.026). → Jošt 2024

Evidence¶

Jošt 2024¶

Jošt, G.; Taneski, V.; Karakatič, S. (2024). The Impact of Large Language Models on Programming Education and Student Learning Outcomes. Appl. Sci. 14, 4115. https://doi.org/10.3390/app14104115

q3 · i? · associational · r1

Correlational analysis of data from a ten-week experiment with thirty-two second-year undergraduates learning React; one-sided Spearman correlation with bootstrap CIs. The article reports "Spearman's rho = −0.347, p = 0.026"; no standardized effect size beyond rho is printed.

"We identified a significant negative correlation between the average use of LLMs and students' final grades (Spearman's rho = −0.347, p = 0.026)"

Discussion¶

Related Claims¶

• LLM use for debugging is significantly negatively correlated with student final grades — a narrower finding that bears on this claim
• LLM use for code generation is significantly negatively correlated with student final grades — a narrower finding that bears on this claim
• LLM use for seeking additional explanations does not significantly impact final grades — a narrower finding that bears on this claim
• A downward trend in final grades is observed with increased average LLM use across all task types — related