Demystify, use, and reflect: a structured LLM-intervention course design positioning LLMs as tools for supported learning

Design · All designs
Evidence · 3 claims (3 for) · 1 study (1 associational), q3 · 0 of 1 report an effect size · 3 claims rest on one study

Description

The course design centers on a structured intervention that works by "demystifying LLMs, how they work, their capabilities, and their probabilistic nature and limitations, alongside guided use through prompting strategies and iterative interaction". Rather than mandating or banning LLM use, the course leaves the decision to each student, requires students to understand, explain, and acknowledge AI-generated contributions, and adds a post-assignment reflection survey capturing prompt counts, role conceptualization, and verification strategies. This study extends the intervention by examining how its effects vary at the subfield level rather than only in aggregate.

Design Implications

Context

Requirements

• Students need access to an LLM tool (institutionally licensed Microsoft Copilot Chat was available, though not restricted to a single tool)
• Post-assignment reflection surveys must be administered immediately after task completion to capture interaction

Constraints

• The course did not explicitly encourage LLM use; the decision of whether and how to use them was left to each student
• Findings are tied to the specific assignments and instructional practices of a single course context

Target Learners

• Undergraduate computer science students in a required post-CS1 problem-solving course

Learning Goals

• Informed and responsible LLM use
• Effective prompting and iterative interaction
• Assessing the reliability of LLM-generated responses

Claims

• Llm Adoption Varies By Cs Assignment [+M]
• Assistant Role Dominates Llm Conceptualization [+M]
• Verification Nearly Universal Multiple Strategies [+M]

Related Designs

Examples

-

Key Sources

• Nizamani, S. B., Lee, Y., Donekal Chandrashekar, N., Ellis, M., & Ramakrishnan, N. (2026). Understanding Student Use of Large Language Models Across Computer Science Subfields. https://arxiv.org/abs/2610.01158. doi:10.48550/arxiv.2610.01158

Page metadata

Field	Value
type	design
id	demystify-use-reflect-llm-intervention
title	Demystify, use, and reflect: a structured LLM-intervention course design positioning LLMs as tools for supported learning
description	The course design centers on a structured intervention that works by "demystifying LLMs, how they work, their capabilities, and their probabilistic nature and limitations, alongside guided use through prompting strate...
status	review
generated	by process:settle-candidates on 2026-10-11
trust	unverified — no one has confirmed this page's content
sources	1 — nizamani-2026

Frontmatter is the page's machine-readable record — see the Schema & Guide.