A task-specific prompt with explicit guidance about correct-answer cases significantly outperforms literature-based PedCoT prompting on TM test cases (84% vs 59% accuracy)¶

Claim · All claims
Evidence · 1 study · causal r? · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The task-specific prompt significantly outperformed literature-based PedCoT prompting on the 61 TM test cases (84% vs 59% accuracy, McNemar's p <0.01). → Moiz Imran and Sahan Bulathwela 2026

Evidence¶

Moiz Imran and Sahan Bulathwela 2026¶

Moiz Imran and Sahan Bulathwela. (2026). Catching TheCorrectAnswerTrap: Characterising AI Tutor Blind Spots When Analysing Student Reasoning. arXiv preprint. https://arxiv.org/abs/2605.23925 doi:10.48550/arxiv.2605.23925

q2 · i? · causal · r?

Prompt validation experiment comparing the paper's task-specific prompt against PedCoT literature-based prompting on the 61 TM test cases. The task-specific prompt reached 84% versus 59% accuracy, significant by McNemar's test (p <0.01).

"We validated this prompt against PedCoT [11] on the 61 TM test cases, finding our task-specific prompt significantly outperforms literature-based prompting (84% vs 59% accuracy, McNemar'sp <0.01)."

Discussion¶

Related Claims¶

• Frontier LLMs achieve the best balance between detecting misconceptions in correct-answer cases and not over-flagging correct students, while fine-tuned local models achieve under 58% TM recall — related
• Fine-tuned classifiers detect only about 57% of correct-answer misconceptions despite near-perfect wrong-answer classification — related
• Including a mark scheme in the prompt does not change misconception detection but significantly reduces false positives against correct-reasoning students — related
• Correct-answer-trap cases concentrate in specific question types where common flawed reasoning happens to produce the correct numerical answer, with the top two question types accounting for 71% of all TM cases — related
• Enforcing an explicit multi-step reasoning structure yields no significant improvement for the two studied models, which already self-organize an effective reasoning structure — related
• Accuracy differences between the top efficient models were not statistically significant at the high school level but were significant at the middle school level, with Nemotron scoring better — related
• On the PRM800K validation dataset, detection rates dropped by 10.5 percentage points when final answers were correct, consistent with the correct answer trap extending beyond the Eedi data — related