Higher-struggling-level problems elicit more diverse but less student-like LLM-generated errors¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Mean edit distances among LLM submissions increased substantially from low- to medium-struggling problems and remained high at high-struggling problems, indicating harder problems elicit more heterogeneous erroneous outputs. → Ali Keramati 2026 q2 i? As struggling level increases, mean nearest edit distance to human code also increases, indicating LLMs find it harder to reproduce student-like logical errors on higher-struggle problems. → Ali Keramati 2026 (2)

Evidence¶

Ali Keramati 2026¶

Ali Keramati, Jie Cao, Iman Mohammadi, Mark Warschauer, Yang Shi. (2026). Simulating Students' Java Programming Errors with Large Language Models. arXiv preprint. https://arxiv.org/abs/2606.14113 doi:10.48550/arxiv.2606.14113

q2 · i? · design · r2

Figure-based comparison of diversity across three struggling levels (Low avg. 1,187 submissions, Medium avg. 2,049, High avg. 4,125). Example: Gemini 2.5 Pro (CoT) rose from 62.34 (low) to 82.89 (medium) and stayed high at 74.59 (high).

"mean edit distances among LLM submissions increased substan- tially from low- to medium-struggling problems, and re- mained high at high-struggling problems for most models"

Ali Keramati 2026 (2)¶

Ali Keramati, Jie Cao, Iman Mohammadi, Mark Warschauer, Yang Shi. (2026). Simulating Students' Java Programming Errors with Large Language Models. arXiv preprint. https://arxiv.org/abs/2606.14113

q2 · i? · design · r2

Figure 4 analysis of simulated-human alignment across struggling levels. Claude Sonnet 4 was most stable with slightly longer distances as struggle increased; GPT-5 and Gemini 2.5 Pro showed substantial increases at medium and high levels.

"as the level of struggle increases, the mean near- est edit distance also increases, indicating that LLMs find it harder to reproduce student-like logical errors on higher- struggle problems"

Discussion¶

Related Claims¶

• LLM-generated erroneous code aligns with authentic student errors at levels that depend on model and prompting strategy, with Claude Sonnet 4 most consistently aligned — related
• LLMs across five models generate diverse erroneous Java code, with mean AST edit distances varying by model and prompting strategy — related
• LLM error simulation rarely fails, but the model is less likely to reproduce a targeted distractor on problems with longer correct solutions — related
• On tightly specified functions, LLMs converged on a few exact AST forms (1–14 per 1,000 files) while an open-ended function remained diverse, and source was associated with exact form for all five functions — related