INSIDE fine-tuning achieves the highest alignment between generated internal dialogue and ground-truth student code edits, up to 57.9%¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? On test_NP, Qwen3-8B-INSIDE reaches 57.9% alignment versus 56.0% for the strongest BloomCoT prompting model; on test_OP, Qwen2.5-7B-INSIDE reaches 51.8% versus 49.6% for the best prompting baseline. → Rose Niousha 2026 q2 i? The LLM-judge alignment metric was validated against human annotations, achieving 88.0% agreement (kappa = 0.754). → Rose Niousha 2026 (2)

Evidence¶

Rose Niousha 2026¶

Rose Niousha, Minwoo Kang, Narges Norouzi. (2026). INSIDE the Student's Mind: Jointly Modeling Latent Reasoning and Action in LLM Student Simulators. Published as a conference paper at COLM 2026. https://arxiv.org/abs/2608.10492 doi:10.48550/arxiv.2608.10492

q2 · i? · design · r2

Evaluation of Experiment 2 models on test_NP (1,054 submissions) using an LLM judge that decomposes generated internal dialogue into atomic claims checked against ground-truth code diffs. The article reports Qwen3-8B-INSIDE "achieves 57.9% compared to 56.0% for the strongest BloomCoT model"; test_NP has larger standard errors (~2.0-2.3).

"Ontest_NP, Qwen3-8B-INSIDE achieves 57.9% compared to 56.0% for the strongest BloomCoT model."

Rose Niousha 2026 (2)¶

Rose Niousha, Minwoo Kang, Narges Norouzi. (2026). INSIDE the Student's Mind: Jointly Modeling Latent Reasoning and Action in LLM Student Simulators. Published as a conference paper at COLM 2026. https://arxiv.org/abs/2608.10492

q2 · i? · design · r2

Validation study of the alignment metric: 25 generated internal dialogues were manually annotated for whether each extracted claim was reflected in the corresponding code edit. The article reports "88.0% agreement with the LLM judge labels ( κ= 0.754)"; a separate teacher-trace check (n=209) yielded 95.2% average alignment.

"The manual annotations achieved 88.0% agreement with the LLM judge labels ( κ= 0.754), indicating substantial agreement between the LLM judge and human annotations."

Discussion¶

Related Claims¶

• Prompting-based student simulators maintain artificially high pass rates (~80%) from the first submission step, while fine-tuned models track real students' low-start, sharp-increase trajectories — related
• Fine-tuning with INSIDE internal dialogue achieves the lowest Wasserstein distances to real student code distributions on test_OP across all action-fidelity metrics — related
• High self-consistency between a simulator's reasoning and its own generated code does not imply alignment with real student reasoning or behavior — related
• Differences in student pass-rate distributions between test splits explain the smaller INSIDE-over-SFT improvement on test_NP — related
• LLM-generated erroneous code aligns with authentic student errors at levels that depend on model and prompting strategy, with Claude Sonnet 4 most consistently aligned — related