Under a realistic estimator, remaining error originates upstream rather than from the distillation step¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? When the X-learner replaces the oracle, the mentee's correlation with truth falls to about 0.72 and slope to about 0.50, but the paired mentor shows comparable compression, with every mentee–mentor gap at most 0.074. → Chenguang Pan 2026

Evidence¶

Chenguang Pan 2026¶

Chenguang Pan, Airui Meng, and Youmi Suk. (2026). Distilling Black-Box Machine Learning into a Small, Self-Explaining Language Model for Learning Analytics. arXiv. https://arxiv.org/abs/2608.21165 doi:10.48550/arxiv.2608.21165

q2 · i? · design · r2

Simulation study switching the upstream estimator from oracle CATE to X-learner. "The performance degradation can be clearly explained by the metrics of the paired mentor." The mentee's point estimation is actually closer to ground truth than the mentor's (r = 0.723 vs. 0.654), attributed to ALE decomposition and LLM distillation smoothing the noisy CATE.

"The performance degradation can be clearly explained by the metrics of the paired mentor, whose own slope against the truth is0.535and0.539, so the compression is present in the estimator before any narration is written, and every mentee–mentor gap in these two columns is at most0.074."

Discussion¶

Related Claims¶

• Distillation step is nearly lossless when the mentor signal is an oracle — related
• Decision quality collapses toward the majority action in severely imbalanced settings — related
• Mentee fidelity on HSLS:09 test split: attribution transmits far better than decisions, and 98.8% of narrations pass the faithfulness audit — related
• Narration fluency is no evidence of correctness: narration quality is independent of signal quality — related