LLMs across five models generate diverse erroneous Java code, with mean AST edit distances varying by model and prompting strategy¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? All five evaluated LLMs produce non-trivial mean edit distances, indicating diverse erroneous code, with Gemini 2.5 Pro and Claude Sonnet 4 most diverse and GPT-4o least. → Ali Keramati 2026

Evidence¶

Ali Keramati 2026¶

Ali Keramati, Jie Cao, Iman Mohammadi, Mark Warschauer, Yang Shi. (2026). Simulating Students' Java Programming Errors with Large Language Models. arXiv preprint. https://arxiv.org/abs/2606.14113 doi:10.48550/arxiv.2606.14113

q2 · i? · design · r2

Automated AST analysis (Zhang-Shasha edit distance) of erroneous code generated by five LLMs under IO, CoT, and Self-Refine prompting for 37 Java problems. The study reports "non-trivial mean edit distances observed across all models and prompting techniques"; Gemini 2.5 Pro and Claude Sonnet 4 exceeded 60 under most prompts, GPT-4o ranged around 25-40.

"the results in Table 1 indicate that LLMs (across five models) are capable of producing diverse erroneous code, as reflected in the non-trivial mean edit distances observed across all models and prompting techniques"

Discussion¶

Related Claims¶

• Higher-struggling-level problems elicit more diverse but less student-like LLM-generated errors — related
• LLM-generated erroneous code aligns with authentic student errors at levels that depend on model and prompting strategy, with Claude Sonnet 4 most consistently aligned — related
• On tightly specified functions, LLMs converged on a few exact AST forms (1–14 per 1,000 files) while an open-ended function remained diverse, and source was associated with exact form for all five functions — related