In a ten-request live textbook audit, citation integrity coexisted with substantive error: every citation resolved to its source, yet answers included retrieval misses and contradictions¶

Claim · All claims
Evidence · 1 study · design r3 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? A targeted ten-request live textbook audit found three supported replies, four retrieval misses despite available evidence, two source contradictions, and one partly grounded answer, with every returned citation resolving to its source. → Sidney Shapiro and Joshua Lindemann 2026

Evidence¶

Sidney Shapiro and Joshua Lindemann 2026¶

Sidney Shapiro and Joshua Lindemann. (2026). On-Premises Multi-Course RAG Tutoring for Business Education: Hardware–Software Trade-offs in a Campus AI Tutor. arXiv. https://arxiv.org/abs/2610.02510 doi:10.48550/arxiv.2610.02510

q2 · i? · design · r3

A targeted, agent-reviewed ten-request live textbook audit of the production path; the section reports "three supported replies, four retrieval misses despite available evidence" and notes every returned citation resolved to its source while meaning was not always preserved.

"It found three supported replies, four retrieval misses despite available evidence, two source contra- dictions, and one partly grounded answer."

Discussion¶

Related Claims¶

• Of 100 human-audited claims, 39 are fully verifiable, 55 partially verifiable, and 6 not verifiable, with main responses showing stronger provenance than follow-ups — related
• Fifth-grade history textbook excerpts largely fail to answer the frame slot questions, presenting detailed action but missing main ideas — related
• AI output errors are frequent: 44.5% fabricated references in drafted research proposals and 5–13% unsafe chatbot medical answers — related
• The platform's out-of-scope audit protocol is designed to neutralize student distrust of AI error by rewarding verified hallucination reports — related
• A git-history audit found 96.3% of AI-generated answer-key question blocks needed no post-release semantic repair — related
• Checking can coexist with unresolved errors: 7 of 15 students failed a task containing plausible LLM falsehoods despite self-checking — related
• Retrieval failure explained only 2 of vector RAG's 11 ungrounded answers; 9 answers retrieved the correct material and were still ungrounded, suggesting chunked context invites fabrication — related
• NLI analysis reveals reference-based grading is limited: many semantically correct responses contradict reference answers — related
• Post-deployment diagnostics attributed many remaining retrieval misses to query formulation, with bounded expansion recovering eleven of 24 original queries without lowering the relevance floor
• Llama Guard models misclassify subtle, context-dependent unsafe education prompts (e.g., exam-answer and cheating requests) as safe — related
• Manual error analysis finds incomplete retrieval is the most frequent failure category, ahead of code-semantics limits, bilingual retrieval gaps, and over-conservative refusals — related