GRPO with an SFS-aligned reward yields more consistent improvements than DPO, achieving the highest SFS overall¶

Claim · All claims
Evidence · 1 study · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across three of four settings, GRPO increases targeted-feedback flip rate while maintaining or reducing misaligned flip rate relative to SFT, achieving the highest SFS overall; DPO produces only modest changes over SFT. → Heejin Do 2026

Evidence¶

Heejin Do 2026¶

Heejin Do, Shashank Sonkar, Mrinmaya Sachan. (2026). Simulating Students or Sycophantic Problem Solving? On Misconception Faithfulness of LLM Simulators. Preprint. https://arxiv.org/abs/2605.12748 doi:10.48550/arxiv.2605.12748

q2 · i? · causal · r2

Post-training comparison of DPO and GRPO on Qwen3-4B and Llama3.1-8B-Instruct across Malrule and EEDI (Figure 5); the text states GRPO "achieving the highest SFS overall" in three of four settings.

"GRPO, by contrast, yields more consistent improvements: across three of four settings, GRPO increasesFT while maintaining or reducingFM relative to SFT, achieving the highest SFS overall."

Discussion¶

Related Claims¶

• Supervised fine-tuning on misconception-faithful demonstrations sharply improves SFS (up to +0.555 on Malrule for Qwen3-4B) — related
• LLM student simulators exhibit near-zero Selective Flip Score, flipping at similarly high rates under targeted, misaligned, and generic feedback — related
• SFS-aligned post-training generalizes out-of-domain: models trained on one dataset retain substantial SFS gains on the other — related
• Sycophantic failure manifests as indiscriminate correct flips rather than explicit sycophantic agreement, and SFT shifts behavior toward pushback and maintain under non-targeted feedback — related
• Applying GRPO alone to LLMs for tagging yields significantly lower performance than supervised fine-tuning — related
• GRPO with the Proficiency Alignment Reward improves alignment over SFT across every writing trait and essay prompt — related
• Taxonomy-guided post-training raises TACTutor's TACT_Overall on TACTBench from 0.629 (backbone) to 0.832, the highest among evaluated models — a narrower finding that bears on this claim
• Post-training alignment methods may induce sycophancy that deteriorates GenAI models' ability to disagree and push back — related
• General-purpose LLMs show sycophancy: AI responses affirmed users substantially more than human responses, increasing trust and reducing willingness to take responsibility — related