In a real programming course, students with an eliciting LLM agent on a learning dashboard wrote more reflection messages than students with a telling agent¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Students in the elicit condition produced significantly more reflection-category messages than students in the tell condition (z = 2.47, p = 0.013). → Graf 2026

Evidence¶

Graf 2026¶

Graf, L., Bassner, P., Anzinger, M., Dietrich, F., Krusche, S., & Poquet, O. (2026). Interactive learning dashboards: rethinking learning visualisations as engagement tools. Education and Information Technologies. https://doi.org/10.1007/s10639-026-14082-1

q2 · i? · causal · r2

Exploratory case study: open bottom-up coding of a random sample of learner–LLM messages (over 300 messages, Cohen's Kappa = 0.8) compared reflection-category frequency between the elicit and tell conditions. The z-test found "a z-statistic of 2.47 and a p-value of 0.013".

"A two-proportion z-test was conducted to clarify if the difference in the frequency of a reflection between the tell and elicit condition was statistically significant (p < 0.05), revealing a z-statistic of 2.47 and a p-value of 0.013."

Discussion¶

Related Claims¶

• Reflective Practice Improves Outcomes When Structured — the general claim this finding bears on, with the studies behind it
• Most students submitted JoLs out of curiosity about the system rating, often without attending to the LA graphs, and reported behaviour change more from JoL than from the agent — related
• JoL–mastery accuracy trajectories over the five weeks rose in the elicit condition, declined in the none condition, and were mixed in the tell condition — related
• Tell and no-agent conditions showed no significant JoL–confidence correlations, with mixed results for progress and mastery — related
• Students using the eliciting agent showed JoL ratings correlated with system learning metrics, strongest for mastery — related