No consistent relationship was observed between false-positive and false-negative rates and either the question or the LLM model¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? The study did not observe a consistent relationship between relative rates of false positives and false negatives and either the specific question or LLM model, contrary to expectations of systematic leniency or strictness. → Jacob Levine 2026

Evidence¶

Jacob Levine 2026¶

Jacob Levine, Miguel Aenlle, Craig Zilles, Matthew West, and Mariana Silva. (2026). Automated Grading of Handwritten Mathematics Using Vision-Capable LLMs. https://arxiv.org/abs/2605.19043 doi:10.48550/arxiv.2605.19043

q2 · i? · design · r2

Null observational finding from the rubric-item-level error analysis across five questions and three models. The article reports no consistent relationship was observed; the authors had expected some models might be consistently more lenient or strict. No effect size or test statistic is printed.

"Interestingly,wedidnotobserveaconsistentrelationshipbetweentherelativerates of false positives and false negatives and either the specific question or LLM model."

Discussion¶

Related Claims¶

• LLM-based scoring produced fewer false negatives than Deep Tutor on a balanced test set, which the authors expect would spare students the frustration of correct explanations judged incorrect — related
• Some LLMs exhibit a selection bias against selecting option D on multiple-choice pedagogy questions — related
• GPT models showed lower accuracy with more rubric-related errors (28% vs. 13% for Gemini 3 Flash) and near parity between false positives and negatives — related
• AI grading errors concentrate in graphical tasks and include both false positives on incorrect equations and false negatives from misread sketches and labels — related
• Small language models used as automated judges exhibit severe leniency bias against tutoring responses — reports the opposite
• Validator false-negative cases reveal structural reasoning weaknesses, including answer-schema misinterpretation and internal inconsistency — related