Differences in student pass-rate distributions between test splits explain the smaller INSIDE-over-SFT improvement on test_NP¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? test_OP students fail at 55% and succeed at 28% (roughly 2:1), whereas test_NP students fail at 47% and succeed at 34% (roughly 1.4:1); where failures are more common, INSIDE has more mismatch over SFT to correct. → Rose Niousha 2026

Evidence¶

Rose Niousha 2026¶

Rose Niousha, Minwoo Kang, Narges Norouzi. (2026). INSIDE the Student's Mind: Jointly Modeling Latent Reasoning and Action in LLM Student Simulators. Published as a conference paper at COLM 2026. https://arxiv.org/abs/2608.10492 doi:10.48550/arxiv.2608.10492

q2 · i? · design · r2

Distributional analysis (Appendix D, Figure 4) of exact pass-rate mass by split for student, SFT, and INSIDE submissions. The article reports both splits concentrate mass at pass rates 0 and 1 (83% and 81%), and that on test_OP SFT underestimates complete failure (43% vs. 55%) and overestimates complete success (40% vs. 28%), which INSIDE better corrects.

"test_OP students fail at 55% and succeed at 28% (roughly 2:1), whereas test_NP students fail at 47% and succeed at 34% (roughly 1.4:1). This helps explain the smaller INSIDE–SFT gap on test_NP."

Discussion¶

Related Claims¶

• Prompting-based student simulators maintain artificially high pass rates (~80%) from the first submission step, while fine-tuned models track real students' low-start, sharp-increase trajectories — related
• Fine-tuning with INSIDE internal dialogue achieves the lowest Wasserstein distances to real student code distributions on test_OP across all action-fidelity metrics — related
• INSIDE fine-tuning achieves the highest alignment between generated internal dialogue and ground-truth student code edits, up to 57.9% — related
• Pass@1 is not a sufficient success metric for soft barriers because it conflates successful barriers with model generation failures — related
• GPT-generated feedback passes simulated-student helpfulness evaluations only about 35% of the time on real student data — related
• High self-consistency between a simulator's reasoning and its own generated code does not imply alignment with real student reasoning or behavior — related