Edu-QuRaters recover held-out GPT-4.1-mini pairwise educational preferences from single-text scores, with mean accuracy rising from 0.895 (Sheared-LLaMA-1.3B) to 0.917 (Gemma-3-4B-PT)¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Across two base models and six criteria, trained Edu-QuRaters recover held-out pairwise preferences with both base models exceeding 0.86 accuracy on every criterion, and Gemma-3-4B-PT improving accuracy uniformly across every criterion. → Garrod 2026

Evidence¶

Garrod 2026¶

Garrod, O. G. B., Ince, R. A. A., Liu, M., Huti, M., Boos, M., Waldock, A., Andrews, D., Alonso-Kropil, R., & Atherton, P. (2026). Edu-QuRating: Multi-Dimensional Educational Data Curation with Distilled Pairwise Judgements. arXiv. https://arxiv.org/abs/2609.09425 doi:10.48550/arxiv.2609.09425

q2 · i? · design · r2

Numerical evaluation on 50,000 held-out GPT-4.1-mini pairs per criterion, where each Edu-QuRater scores texts independently and the sign of the score difference determines the implied winner. The article reports "Mean accuracy rises from 0.895 for Sheared-LLaMA-1.3B to 0.917 for Gemma-3-4B-PT."

"Across the two base models and six criteria, Edu-QuRaters recover the held-out pairwise preferences with high accuracy (Figure 2). Both base models exceed 0.86 accuracy on every criterion. Mean accuracy rises from 0.895 for Sheared-LLaMA-1.3B to 0.917 for Gemma-3-4B-PT."

Discussion¶

Related Claims¶

• Core-Ed Edu-QuRater scores vary sensibly with education-level metadata on out-of-sample materials (marginal Pearson correlations 0.62 and 0.64; six-dimension cross-validated R2 of 0.449) — related
• In held-out pairwise judge evaluation, combining Edu-QuRater and answer-structure GRPO rewards yields the highest pedagogical-quality win rate (81.08%) while retaining instruction-following above parity (68.24%) against the Qwen3-4B base model — related
• Distillation validation loss decreases sharply from 20k to 200k pairwise examples and plateaus beyond 200k for the Sheared-LLaMA-1.3B base model — related
• The pairwise-distillation procedure transfers to foundational-literacy scoring, with Gemma-3-4B-PT validation losses of 0.128 (student-facing) and 0.175 (teacher-facing) — related
• Edu-QuRater rewards alone raise pedagogical-quality win rate (77.70%) but fall below parity on instruction following (40.54%), consistent with over-optimizing a proxy reward — related
• In matched single-run 1B pre-training comparisons, all Edu-QuRating-based mixtures reach higher observed aggregate accuracy at 30k steps than the FineWeb-Edu baseline, with gains concentrated in specific tasks — related