High self-consistency between a simulator's reasoning and its own generated code does not imply alignment with real student reasoning or behavior¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Prompting-based models achieve high self-consistency (86.9%-99.0%), with GPT-5 highest, yet GPT-5 performs among the worst at reproducing realistic student code trajectories. → Rose Niousha 2026

Evidence¶

Rose Niousha 2026¶

Rose Niousha, Minwoo Kang, Narges Norouzi. (2026). INSIDE the Student's Mind: Jointly Modeling Latent Reasoning and Action in LLM Student Simulators. Published as a conference paper at COLM 2026. https://arxiv.org/abs/2608.10492 doi:10.48550/arxiv.2608.10492

q2 · i? · design · r2

Self-consistency evaluation (Table 4) measuring alignment between each model's generated internal dialogue and its own generated code diff, on test_OP and test_NP. The article concludes that "strong consistency between a model's reasoning and its own generated code does not necessarily imply alignment with real student reasoning or behavior," since GPT-5 is among the worst on action fidelity. INSIDE achieves 83.0%-87.3%.

"Prompting-based models achieve high self-consistency scores (86.9%–99.0% across both test sets), with GPT-5 attaining the highest scores."

Discussion¶

Related Claims¶

• INSIDE fine-tuning achieves the highest alignment between generated internal dialogue and ground-truth student code edits, up to 57.9% — related
• Fine-tuning with INSIDE internal dialogue achieves the lowest Wasserstein distances to real student code distributions on test_OP across all action-fidelity metrics — related
• Curricular chain-of-thought prompting improves competency-classification accuracy over zero-shot, with gains concentrated in larger models, while definition-based prompting does not improve performance — related
• Beam search-based self-refinement improves simulated solution alignment with predicted student behavior — related
• Behavior-prediction accuracy rises with more past learning records, reaching 0.94 at 40 records — related
• Prompting-based student simulators maintain artificially high pass rates (~80%) from the first submission step, while fine-tuned models track real students' low-start, sharp-increase trajectories — related
• Differences in student pass-rate distributions between test splits explain the smaller INSIDE-over-SFT improvement on test_NP — related
• LLM-generated erroneous code aligns with authentic student errors at levels that depend on model and prompting strategy, with Claude Sonnet 4 most consistently aligned — related