NLI analysis reveals reference-based grading is limited: many semantically correct responses contradict reference answers¶

Claim · All claims
Evidence · 1 study · design r2 · q1
Currency · newest study 2025 · checked against its sources 2026-10-19 (automated, 1 of 1 entry) · not reviewed by a person · recheck by 2028: tool and model accuracy changes quickly

Subclaims¶

q1 i? Natural Language Inference analysis found that many student responses contradicted reference answers despite being semantically correct, highlighting limits of reference-based grading. → Pecuchova 2025

Evidence¶

Pecuchova 2025¶

Pecuchova, J., Benko, Ľ., & Drlik, M. (2025). Automated Grading of Open-Ended Questions in Higher Education Using GenAI Models. International Journal of Artificial Intelligence in Education. https://doi.org/10.1007/s40593-025-00517-2

q1 · i? · design · r2

Authors' NLI analysis of student responses versus reference answers in the software engineering dataset; it showed "many student responses contradicted reference answers despite being semantically correct".

"The study also highlights the limitations of reference-based grading approaches, as shown by the Natural Language Inference analysis, which found that many student responses contradicted reference answers despite being semantically correct."

Discussion¶

Related Claims¶

• In a ten-request live textbook audit, citation integrity coexisted with substantive error: every citation resolved to its source, yet answers included retrieval misses and contradictions — related
• Retrieval failure explained only 2 of vector RAG's 11 ungrounded answers; 9 answers retrieved the correct material and were still ungrounded, suggesting chunked context invites fabrication — related
• Evidence tracing links 96 of 114 generated claims to source chunks, but three claims with retrieval scores of 0.50 or below show retrieval similarity alone is insufficient — related