ITAP generated a full token-edit hint chain to a correct solution for 98.3% of semantically incorrect student states in submission order¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2015 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Running on 294 semantically incorrect states from 15 students, ITAP produced a complete hint chain to a correct final solution for 98.3% of states, with failures concentrated in one problem. → Kelly Rivers & Kenneth R. Koedinger 2015

Evidence¶

Kelly Rivers & Kenneth R. Koedinger 2015¶

Kelly Rivers & Kenneth R. Koedinger. (2015). Data-Driven Hint Generation in Vast Solution Spaces: a Self-Improving Python Programming Tutor. International Artificial Intelligence in Education Society. https://doi.org/10.1007/s40593-015-0070-z

q2 · i? · design · r2

Technical evaluation on a dataset of 585 program states from 15 students in an introductory programming course, of which 294 were syntactically correct but semantically incorrect. ITAP generated full chains for "98.3 % of the solution states (289/294)"; all five failures came from one problem, onlyTwoTrue.

"For 98.3 % of the solution states (289/294), ITAP was able to generate a full token-edit hint chain to a correct final solution. All five of the chains which could not generate a full path to a correct solution"

Discussion¶

Related Claims¶

• Canonical states shorten hint chains on average (6.00 vs 7.61 edits), though abstract chains were longer for some problems — related
• ITAP's path success is robust to state ordering: 99.3% of states generated complete paths across 20 randomized-order runs — related
• ITAP's average hint chain length (5.62 token edits) is shorter than the token length of the teacher solution (8.27), suggesting personalized paths — related
• ITAP's hint generation improves as solution data accumulates over time, supporting its self-improving design — related