Distillation step is nearly lossless when the mentor signal is an oracle¶

Claim · All claims
Evidence · 1 study · design r2 · q2 · i3 large
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i3 Under an oracle mentor, the fine-tuned 2-billion-parameter LLM recovers the true CATE surface with Pearson r = 0.925 and r = 0.910 in the two decision distributions, and ranks the true moderators perfectly (AUC = 1.000), never citing a spurious covariate. → Chenguang Pan 2026

Evidence¶

Chenguang Pan 2026¶

Chenguang Pan, Airui Meng, and Youmi Suk. (2026). Distilling Black-Box Machine Learning into a Small, Self-Explaining Language Model for Learning Analytics. arXiv. https://arxiv.org/abs/2608.21165 doi:10.48550/arxiv.2608.21165

q2 · i3 · design · r2

Simulation study comparing oracle mentor to X-learner mentor in a 2-by-2 design of imbalanced case by oracle/X-learner CATE, computed against ground-truth CATE on a held-out test set over five repetitions. The printed effect sizes are r = 0.925 and r = 0.910 between mentee estimates and true CATE. The mentee also achieves moderator AUC = 1.000 and FalseCite = 0.000 under oracle conditions.

"Pearson correlations reach0.925and0.910, and slopes reach0.902and0.918in the two decision distributions, so the E2B Gemma model fine-tuned on roughly seven thousand narrations both ranks students correctly and reproduces the scale of the effects"

Discussion¶

Related Claims¶

• Under a realistic estimator, remaining error originates upstream rather than from the distillation step — related
• Mentee fidelity on HSLS:09 test split: attribution transmits far better than decisions, and 98.8% of narrations pass the faithfulness audit — a narrower finding that bears on this claim
• Narration fluency is no evidence of correctness: narration quality is independent of signal quality — related
• Decision quality collapses toward the majority action in severely imbalanced settings — related