LLM student simulators exhibit near-zero Selective Flip Score, flipping at similarly high rates under targeted, misaligned, and generic feedback¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · checked against its sources 2026-10-21 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across seven LLMs (4B–120B), two datasets, and base/reflective prompting, SFS remains close to zero, with targeted feedback producing almost no additional behavioral change beyond contrastive controls. → Heejin Do 2026

Evidence¶

Heejin Do 2026¶

Heejin Do, Shashank Sonkar, Mrinmaya Sachan. (2026). Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators. Preprint. https://arxiv.org/abs/2605.12748 doi:10.48550/arxiv.2605.12748

q2 · i? · causal · r2

Simulation-based evaluation of seven instruction-tuned LLMs on Malrule and EEDI misconception-contrastive datasets under base and reflective prompts; Table 3 reports near-uniform flip rates (e.g., Qwen3-4B SFS +0.01) showing "Targeted feedback produces almost no additional behavioral change beyond the contrastive controls."

"Across all models, datasets, and prompting strategies, SFS remains consistently close to zero (Tables 3). Targeted feedback produces almost no additional behavioral change beyond the contrastive controls."

Discussion¶

Related Claims¶

• GRPO with an SFS-aligned reward yields more consistent improvements than DPO, achieving the highest SFS overall — related
• Supervised fine-tuning on misconception-faithful demonstrations sharply improves SFS (up to +0.555 on Malrule for Qwen3-4B) — related
• Sycophantic failure manifests as indiscriminate correct flips rather than explicit sycophantic agreement, and SFT shifts behavior toward pushback and maintain under non-targeted feedback — related
• Stronger problem-solving capability predicts lower misconception faithfulness: flip rate under generic feedback correlates negatively with SFS (Pearson r = -0.94, p = 0.002) — related
• Multi-turn reflective interaction does not restore selective updating and destabilizes weaker simulators — a narrower finding that bears on this claim
• The competence paradox: broadly capable LLMs cannot genuinely unknow expert knowledge when simulating novices — related
• SFS-aligned post-training generalizes out-of-domain: models trained on one dataset retain substantial SFS gains on the other — related