Full reasoning traces impaired task performance relative to both answer-only and summary-trace conditions in AI-assisted LSAT reasoning¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2–r3 · q3 · i2 medium
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i2 Participants solving ten LSAT-style problems with an open-weight verbose-trace model scored significantly worse in the Full-trace condition than in the Answer-only condition (d = -0.54) and the Summary-trace condition (d = -0.48). → Fernandes 2026

Evidence¶

Fernandes 2026¶

Fernandes, D., Buschek, D., Tankelevitch, L., Kosch, T., & Welsch, R. (2026). Explaining Too Much? Understanding How Large Language Model Reasoning Traces Influence Performance and Metacognition. https://arxiv.org/abs/2605.25856 doi:10.48550/arxiv.2605.25856

q3 · i2 · causal · r2

Preregistered between-subjects experiment (N = 559) with three trace-format conditions. The omnibus ANOVA on achieved performance was significant with η² = .052; means were 6.18 (Answer-only), 5.46 (Full-trace), 6.11 (Summary-trace) of ten items.

"Achieved performance differed significantly across conditions (see Table 1 and Figure 4), 𝐹( 2, 556)= 15.10,𝑝<. 001,𝜂2 =. 052."

Fernandes 2026 (2)¶

Fernandes, D., Buschek, D., Tankelevitch, L., Kosch, T., & Welsch, R. (2026). Explaining Too Much? Understanding How Large Language Model Reasoning Traces Influence Performance and Metacognition. https://arxiv.org/abs/2605.25856. doi:10.48550/arxiv.2605.25856

q3 · i2 · causal · r3

Pairwise comparisons from the same experiment show the Full-trace condition performed significantly worse than both named alternatives, with d = -0.54 versus Answer-only and d = -0.48 versus Summary-trace.

"Full-traceparticipants performed significantly worse thanAnswer-only( 𝑡( 368)=− 5.15,𝑝<. 001,𝑑=− 0.54) andSummary-trace( 𝑡( 370)=− 4.64,𝑝<. 001,𝑑=− 0.48)."

Discussion¶

Related Claims¶

• Reasoning-trace exposure raised trust and hedonic appeal without performance benefits, shifting subjective appraisal of the interaction — related
• Participants substantially overestimated their performance in all trace conditions, with the largest overestimation under full traces — related
• Full reasoning traces worsened calibration bias and Brier scores relative to both other conditions — related
• No trace format supported calibrated self-evaluation: perceived-achieved correlations were low and non-significant in all conditions — related
• Summary traces preserved task performance at the answer-only baseline while elevating subjective appraisal — related
• Hedonic appeal, not trust, carries the indirect path from trace exposure to performance overestimation, consistent with a processing-fluency account — related