Claims

CAT concentration is item-driven, not category-driven: removing the two vulnerable items collapses the procedural/conceptual odds ratio from 5.6 to 1.0

Claim · All claims
Evidence · 1 study · associational r? · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims

Reading the evidence codes

q is the design tier, i the impact, n the sample, and the kind with its r the rigour judged within that kind: what each code means.

This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.

q2 i? The vulnerability belongs to specific items with coincidental correctness paths, not to procedural questions as a class; the procedural/conceptual odds ratio drops from 5.6 to 1.0 when the two vulnerable items are removed. → Moiz Imran and Sahan Bulathwela 2026

Evidence

Moiz Imran and Sahan Bulathwela 2026

Moiz Imran and Sahan Bulathwela. (2026). Catching TheCorrectAnswerTrap: Characterising AI Tutor Blind Spots When Analysing Student Reasoning. arXiv preprint. https://arxiv.org/abs/2605.23925 doi:10.48550/arxiv.2605.23925

q2 · i? · associational · r?

Item-level analysis over the 15 unique Eedi questions classified as procedural (n=6), conceptual (n=8) or mixed (n=1). The authors report the odds ratio "drops from 5.6 to 1.0 (p= 0.48)" once the two vulnerable items are removed.

"Removing just these two items, the odds ratio between procedural and conceptual questions drops from 5.6 to 1.0 (p= 0.48). The remaining procedural questions have TM rates no different from conceptual ones."

Discussion

Page metadata
Field Value
type claim
id cat-vulnerability-item-driven-not-category-driven
title CAT concentration is item-driven, not category-driven: removing the two vulnerable items collapses the procedural/conceptual odds ratio from 5.6 to 1.0
description CAT concentration is item-driven, not category-driven: removing the two vulnerable items collapses the procedural/conceptual odds ratio from 5.6 to 1.0
status review
evidence_strength moderate
newest_evidence 2026
recheck_by 2029
generated by process:wiki-ingest on 2026-10-10
trust unverified — no one has confirmed this page's content
sources 1 — moiz-imran-and-sahan-bulathwela-2026

Frontmatter is the page's machine-readable record — see the Schema & Guide.