Chain-of-Thought rationale quality shows no direct correlation with the accuracy of simulated student responses¶

Claim · All claims
Evidence · 1 study · review r2 · q2
Currency · newest study 2025 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? In simulating student answers to exam questions, the quality of CoT-generated rationales did not correlate with the accuracy of simulated responses, though CoT can still be used to simulate flawed reasoning with typical misconceptions. → Luis Marquez-Carpintero 2025

Evidence¶

Luis Marquez-Carpintero 2025¶

Luis Marquez-Carpintero, Alberto Lopez-Sellers, Miguel Cazorla. (2025). Simulating Students with Large Language Models: A Review of Architecture, Mechanisms, and Role Modelling in Education with Generative AI. https://arxiv.org/abs/2511.06078 doi:10.48550/arxiv.2511.06078

q2 · i? · review · r2

The review reports findings from Benedetto et al. [2024] on simulating student responses to exam questions with varying proficiency levels. The article reports no correlation between rationale quality and simulated-response accuracy; no coefficient is printed.

"However, recent findings in the context of simulating student answers to exam questions indicate no direct correlation between the quality of CoT-generated rationales and the accuracy of simulated responses"

Discussion¶

Related Claims¶

• Curricular chain-of-thought prompting improves competency-classification accuracy over zero-shot, with gains concentrated in larger models, while definition-based prompting does not improve performance — related
• Under Chain-of-Thought prompting, higher Bloom-level classification accuracy correlates with lower category consistency in generated questions — related
• Chain-of-thought and reasoning prompting significantly improve LLM proportional match with human-authored distractors over direct prompting on both math and science MCQs — related
• Higher knowledge-identification accuracy is associated with improved knowledge coverage in LLM-generated questions under Chain-of-Thought prompting — related
• Simulating higher-cognitive-ability students is easier: simulation quality correlates positively with student cognitive score — related
• The competence paradox: broadly capable LLMs cannot genuinely unknow expert knowledge when simulating novices — related