LLM-generated erroneous code aligns with authentic student errors at levels that depend on model and prompting strategy, with Claude Sonnet 4 most consistently aligned¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Mean nearest edit distances to human submissions show LLMs can resemble student errors (best 16.35, Claude Sonnet 4 with Self-Refine), but GPT-5 and Gemini 2.5 Pro often exceed 80. → Ali Keramati 2026

Evidence¶

Ali Keramati 2026¶

Ali Keramati, Jie Cao, Iman Mohammadi, Mark Warschauer, Yang Shi. (2026). Simulating Students' Java Programming Errors with Large Language Models. arXiv preprint. https://arxiv.org/abs/2606.14113 doi:10.48550/arxiv.2606.14113

q2 · i? · causal · r2

Two-stage structural alignment analysis comparing each LLM-generated error against 74,080 deduplicated human submissions via Jaccard retrieval and Zhang-Shasha tree edit distance. "Claude Sonnet 4 provides the most consistent alignment"; Grok Code Fast 1 achieved the absolute minimum (16.22) with Self-Refine but fluctuated above 30 under IO and CoT.

"Claude Sonnet 4 provides the most consistent alignment: across all prompting strategies, its distances remain rela- tively low, with the best case (16.35 under Self-refine) be- ing the closest match to real student code among all con- ditions."

Discussion¶

Related Claims¶

• Beam search-based self-refinement improves simulated solution alignment with predicted student behavior — related
• Higher-struggling-level problems elicit more diverse but less student-like LLM-generated errors — related
• LLMs across five models generate diverse erroneous Java code, with mean AST edit distances varying by model and prompting strategy — related
• Blinded expert annotators misclassified 83.7% of LLM-generated errors as human-written — related
• LLM-generated errors received significantly higher plausibility ratings than authentic student errors, explained by a narrower, cleaner error-type distribution — related
• Llama 4 Maverick shows the strongest human-LLM alignment, while Claude Sonnet 4 performs weakest with poor calibration — related
• INSIDE fine-tuning achieves the highest alignment between generated internal dialogue and ground-truth student code edits, up to 57.9% — related
• Fine-tuning with INSIDE internal dialogue achieves the lowest Wasserstein distances to real student code distributions on test_OP across all action-fidelity metrics — related
• High self-consistency between a simulator's reasoning and its own generated code does not imply alignment with real student reasoning or behavior — related
• Behavioral similarity between LLMs and learners does not guarantee mechanistic similarity — related