Students in a scaffolded LLM-collaboration course improved significantly on initial argument quality, prompt engineering, and response to AI feedback from the first to the fifth assignment¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2025 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q2 i? Paired-sample t-tests showed significant improvement on all three graded dimensions (initial argument, prompt engineering, response to feedback) between Assignment 1 and Assignment 5. → Daniel M. Oppenheimer 2025

Evidence¶

Daniel M. Oppenheimer 2025¶

Daniel M. Oppenheimer, Trent N. Cash, Allison E. Connell Pensky. (2025). You've Got AI Friend in Me: LLMs as Collaborative Learning Partners. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-025-00521-6

q2 · i? · design · r2

Paired-sample t-tests comparing each student's Assignment 1 and Assignment 5 scores graded on 3-point scales by a teaching assistant. Means rose from 2.58 to 2.90 (initial argument) and 2.34 to 2.89 (prompt engineering); the article states "Student improvement was equivalent to approximately a full standard deviation on each of the dimensions." No standardized effect size is printed.

"Students showed significant improvement on all three dimensions (see Fig. 1): Initial Argument (x̄ assignment1 = 2.58, x̄ assignment5 = 2.90, t(109) = 6.22, p <. 001), Prompt Engineering (x̄ assignment1 = 2.34, x̄ assignment5 = 2.89, t(109) = 8.80, p <. 001)"

Discussion¶

Learner Variables¶

• Writing Skills — outcome: instruction changes it

Related Claims¶

• Direct GenAI-supported feedback improves immediate argument-quality gain over peer feedback, while reflective and hybrid designs outperform direct GenAI feedback — the general claim this finding bears on, with the studies behind it
• Students became better able to develop arguments even prior to interacting with LLMs, per the authors' interpretation of initial-draft gains — related
• Students’ prompt self-efficacy increased significantly after the scaffolded GenAI course (t = 3.729, p < .001) — related
• Students rated the LLM-based assignments as useful, challenging, engaging, and enjoyable at both the first and fifth iterations, with no significant change across the semester — related
• A 60-minute session with Prompt Coach significantly improved professional developers' prompt quality, a 13.73% relative gain overall — related
• Students' argumentative essay quality improves substantially from initial to revised drafts regardless of feedback source — related
• Students' self-efficacy for using generative AI increased over the semester, with current self-efficacy significantly higher after the fifth assignment than after the first — related
• Peer-to-peer prompts show promising effects on collaborative argumentation, consensus, and perceived efficacy, suggesting new roles for peer learning in classrooms — related
• Higher-quality prompt engineering skills predict the quality of LLM output in goal-directed use of generative AI tools — related