Full reasoning traces worsened calibration bias and Brier scores relative to both other conditions¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r3 · q3 · i1–i2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i1 Calibration bias differed significantly across conditions, with Full-trace showing greater overconfidence (M = 0.265) than Answer-only (M = 0.189). → Fernandes 2026 q3 i2 Brier scores differed significantly across conditions, with Full-trace producing the worst (highest) scores (M = 0.342). → Fernandes 2026 (2)

Evidence¶

Fernandes 2026¶

Fernandes, D., Buschek, D., Tankelevitch, L., Kosch, T., & Welsch, R. (2026). Explaining Too Much? Understanding How Large Language Model Reasoning Traces Influence Performance and Metacognition. https://arxiv.org/abs/2605.25856 doi:10.48550/arxiv.2605.25856

q3 · i1 · causal · r3

Trial-level calibration index (mean confidence minus accuracy) differed across conditions with η² = .029; Tukey HSD showed Full-trace (M = 0.265) more overconfident than Answer-only (M = 0.189, p < .001), with Summary-trace between and not reliably different from either.

"Calibration differed between conditions,𝐹( 2, 556)= 8.31,𝑝<. 001,𝜂2 =. 029."

Fernandes 2026 (2)¶

Fernandes, D., Buschek, D., Tankelevitch, L., Kosch, T., & Welsch, R. (2026). Explaining Too Much? Understanding How Large Language Model Reasoning Traces Influence Performance and Metacognition. https://arxiv.org/abs/2605.25856. doi:10.48550/arxiv.2605.25856

q3 · i2 · causal · r3

Brier score ANOVA was significant with η² = .042; Full-trace (M = 0.342) was reliably worse than both Answer-only (M = 0.289) and Summary-trace (M = 0.296), which did not differ (p = .80).

"Conditions differed reliably,𝐹( 2, 556)= 12.20,𝑝<. 001,𝜂2 =. 042."

Discussion¶

Learner Variables¶

• Metacognition — outcome: instruction changes it

Related Claims¶

• Access to AI advice roughly doubled reported confidence in answers in one experiment — the general claim this finding bears on, with the studies behind it
• Full reasoning traces impaired task performance relative to both answer-only and summary-trace conditions in AI-assisted LSAT reasoning — related
• Participants substantially overestimated their performance in all trace conditions, with the largest overestimation under full traces — related
• No trace format supported calibrated self-evaluation: perceived-achieved correlations were low and non-significant in all conditions — related
• Summary traces preserved task performance at the answer-only baseline while elevating subjective appraisal — related
• Learners are substantially overconfident when evaluating the quality of their self-generated examples — related
• Practice with feedback, regardless of feedback type, improves metacognitive calibration relative to lecture, reducing overconfidence by about 13 percentage points — related
• Reasoning-trace exposure raised trust and hedonic appeal without performance benefits, shifting subjective appraisal of the interaction — related