Ten-week two-phase LLM usage experiment in React programming education

Design · All designs
Evidence · 1 claim (1 mixed) · 1 study (1 associational), q3 · 0 of 1 report an effect size · 1 claim rests on one study

Description

A study design in which thirty-two second-year undergraduates completed four React/TypeScript assignments over nine weeks with unrestricted informal LLM use, followed by a controlled week-ten assignment where LLMs were prohibited and only Google and official React documentation were allowed. LLM usage was measured with five-point Likert questionnaire items on code generation, additional explanations, and debugging, administered after grades were released.

Design Implications

Context

Requirements

• Participants must have foundational web development knowledge but no prior formal exposure to the target technology (React)
• Assistants monitored the controlled phase to ensure adherence to the no-LLM rule

Constraints

• The authors acknowledge the relatively small sample size of thirty-two participants and the ten-week duration may not reflect long-term learning trajectories

Target Learners

• second-year undergraduate students

Learning Goals

• learning React application development
• independent implementation of programming tasks

Claims

• Higher average LLM use across all tasks is significantly negatively correlated with final grades in undergraduate programming education [~W]

Related Designs

• Pair-based, discovery-oriented LLM analysis activity following an eight-week disciplinary module

Examples

-

Key Sources

• Jošt, G.; Taneski, V.; Karakatič, S. (2024). The Impact of Large Language Models on Programming Education and Student Learning Outcomes. Appl. Sci. 14, 4115. https://doi.org/10.3390/app14104115

Page metadata

Field	Value
type	design
id	ten-week-two-phase-llm-react-experiment
title	Ten-week two-phase LLM usage experiment in React programming education
description	A study design in which thirty-two second-year undergraduates completed four React/TypeScript assignments over nine weeks with unrestricted informal LLM use, followed by a controlled week-ten assignment where LLMs wer...
status	review
generated	by process:settle-candidates on 2026-10-12
trust	unverified — no one has confirmed this page's content
sources	1 — jošt-2024

Frontmatter is the page's machine-readable record — see the Schema & Guide.