Narration fluency is no evidence of correctness: narration quality is independent of signal quality¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Narration faithfulness metrics (grounding audit, self-closure, uniqueness) remain near 1 across all simulation conditions regardless of whether the upstream signal is true or noisy, showing that fluent, internally consistent text does not indicate correct predictions. → Chenguang Pan 2026

Evidence¶

Chenguang Pan 2026¶

Chenguang Pan, Airui Meng, and Youmi Suk. (2026). Distilling Black-Box Machine Learning into a Small, Self-Explaining Language Model for Learning Analytics. arXiv. https://arxiv.org/abs/2608.21165 doi:10.48550/arxiv.2608.21165

q2 · i? · design · r2

Simulation study narration faithfulness dimension across all four cells of the 2-by-2 design. "The metrics under narration faithfulness are close to 1." Grounding audit for the mentee reaches 0.998 in all conditions and self-closure reaches 0.995–0.999, even when point estimation is poor (r = 0.72).

"The metrics under narration faithfulness are close to 1, which shows that the mentee learns to write arithmetically closed decompositions and that the non-hallucinated narrations are essentially preserved."

Discussion¶

Related Claims¶

• Mentee fidelity on HSLS:09 test split: attribution transmits far better than decisions, and 98.8% of narrations pass the faithfulness audit — related
• Under a realistic estimator, remaining error originates upstream rather than from the distillation step — related
• Distillation step is nearly lossless when the mentor signal is an oracle — related