Most models struggle to engage with a student's prior debugging attempts even when the iteration history is provided¶
Claim · All claims
Evidence · 1 study · designr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? On the six iterative debugging questions, acknowledges_progression scores ranged from 8% to 67%, indicating most models fail to build on the student's prior attempts. → H. Chad Lane 2026
Evidence¶
H. Chad Lane 2026¶
H. Chad Lane, Bryson Kageler. (2026). CSTutorBench: Benchmarking Small Language Models as Tutors for Block-Based Programming. SLM4ED'26: The 1st Workshop of Small Language Models for Education. https://github.com/InviteInstitute/CSTutorBench doi:10.1145/3770761.3777333
q2 · i? · design · r2
Trial 2 per-criterion scoring of the 6 debugging_iterative questions, judged by Claude Sonnet 4 with hybrid review. Each entry provides 2–3 code snapshots of the student's self-directed fix attempts, and the dedicated criterion evaluates whether the response recognizes and builds on these prior attempts. No effect sizes are printed.
"acknowledges_progression(scored on 6 iterative debugging questions) ranged from 8% to 67%, indicating that most models struggle to engage meaningfully with a student’s prior attempts even when that history is provided as context."
Discussion¶
Related Claims¶
- Hint-not-solution is the most challenging rubric dimension, with scores ranging from 6% to 81% across models — related
- Providing the tutor with student mastery and practice-history context improved engagement and next-item correctness — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
models-struggle-acknowledging-debugging-progression |
title |
Most models struggle to engage with a student's prior debugging attempts even when the iteration history is provided |
description |
Most models struggle to engage with a student's prior debugging attempts even when the iteration history is provided |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — h-chad-lane-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.