Expert Q-matrix coding closely matched DINA-recovered structure, with only 2.7% of item-LO codings changed during validation¶

Claim · All claims
Evidence · 1 study · design r3 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? Of 104 data-suggested Q-matrix modifications, coders adopted 20, changing 2.7% of all possible item-LO codings. → Vy Le 2026

Evidence¶

Vy Le 2026¶

Vy Le, Jayson M. Nissen, Jason W. Morphew, Hua Hua Chang, and Ben Van Dusen. (2026). Mechanics Cognitive Diagnostic: Testing Fine-Grained Learning Objectives in Introductory Physics. https://arxiv.org/abs/2609.09584 doi:10.48550/arxiv.2609.09584

q2 · i? · design · r3

Three researchers independently coded items, and DINA-based analysis of posttest responses suggested modifications that coders reviewed by consensus. "The coders reviewed all 104 suggestions and adopted 20 changes," an overall adoption rate of 19%.

"In total, this validation step changed only 2.7% of all possible item-LO codings (20 of 754). This low overall change rate indicates that the initial expert coding closely matched the structure the DINA model recovered from student response data, and it supports the reliability of the final Q-matrix used in the subsequent CD modeling analyses reported below."

Discussion¶

Related Claims¶

• The combined FCI, FMCE, and EMCS item bank covers all 14 learning objectives with at least six items each, mostly single- or two-LO items — related
• A discrimination index can empirically validate Q-matrix specifications by identifying and replacing misspecified entries in the Q-matrix — related
• Misspecification of theoretical diagnostic classification models and inaccurate Q-matrices impact classification accuracy — a broader claim this one bears on