Validator false-negative cases reveal structural reasoning weaknesses, including answer-schema misinterpretation and internal inconsistency¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? In FN cases the Validator sometimes confused the answer value with the option position, and sometimes its textual analysis affirmed alignment while its final binary classification indicated the opposite. → Xiaojing Duan 2026

Evidence¶

Xiaojing Duan 2026¶

Xiaojing Duan, Frederick Nwanganga, and Chaoli Wang. (2026). CODE-GEN: A Human-in-the-Loop RAG-Based Agentic AI System for Multiple-Choice Question Generation. https://arxiv.org/abs/2604.03926 doi:10.48550/arxiv.2604.03926

q2 · i? · design · r2

Qualitative analysis of SME feedback on FN cases from the expert evaluation. SMEs documented that the Validator sometimes "confused the answer value with the option position"; another recurring issue was internal inconsistency between analysis and final classification. No effect size is printed.

"FN cases exposed structural weaknesses in the Validator's reasoning process. A common pattern involved misinterpretation of answer schemas, where the Validator correctly analyzed the code output but confused the answer value with the option position."

Discussion¶

Related Claims¶

• A reasoning-capable open-weight model detects 84% of hidden misconceptions but at realistic prevalence false alarms outnumber genuine detections roughly 8 to 1 — related
• AI grading errors concentrate in graphical tasks and include both false positives on incorrect equations and false negatives from misread sketches and labels — related
• Three recurring AI failure modes arose in this project-based learning context: plausible-but-incorrect code, missing specialized knowledge, and limited long-term context — related
• Students' moral rationalizations of AI use are typically post hoc and often internally inconsistent — related
• Some LLMs exhibit a selection bias against selecting option D on multiple-choice pedagogy questions — related
• No consistent relationship was observed between false-positive and false-negative rates and either the question or the LLM model — related
• LLM-based scoring produced fewer false negatives than Deep Tutor on a balanced test set, which the authors expect would spare students the frustration of correct explanations judged incorrect — related
• GPT models showed lower accuracy with more rubric-related errors (28% vs. 13% for Gemini 3 Flash) and near parity between false positives and negatives — related