Retrieval failure explained only 2 of vector RAG's 11 ungrounded answers; 9 answers retrieved the correct material and were still ungrounded, suggesting chunked context invites fabrication¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? Retrieval itself was largely successful: 43 of 45 topic-page questions (96%) retrieved a chunk from the question's actual source page in the top 5. → Brian Wright 2026 q2 i? Only 2 of the 11 ungrounded Arm A answers were retrieval misses; the other 9 retrieved the correct page or relevant chunks and were still marked ungrounded, with the answerer adding unsupported detail. → Brian Wright 2026 (2)

Evidence¶

Brian Wright 2026¶

Brian Wright. (2026). Potential for Enhanced Learning in Machine Learning Classes by Using Wiki LLM Indexing. https://arxiv.org/abs/2609.25303 doi:10.48550/arxiv.2609.25303

q2 · i? · causal · r2

Retrieval-tracing analysis of Arm A (vector RAG) in the head-to-head evaluation. The article reports "43 of 45 topic page questions(96%)" retrieved from the correct source page; the two misses pulled in short navigational pages and scored low (3 and 4) while marked ungrounded.

"Retrieval itself was largely successful: 43 of 45 topic page questions(96%)retrievedachunkfromthequestion'sactual source page in the top 5."

Brian Wright 2026 (2)¶

Brian Wright. (2026). Potential for Enhanced Learning in Machine Learning Classes by Using Wiki LLM Indexing. https://arxiv.org/abs/2609.25303

q2 · i? · causal · r2

Ungrounded-answer analysis of Arm A in the same evaluation. The article reports "retrieval failure explains only 2 of the11ungroundedArmAanswers" and that the answerer added unsupported detail even with the right excerpts in context, suggesting chunked context invites fabrication.

"More striking is that retrieval failure explains only 2 of the11ungroundedArmAanswers.Theother9retrievedthe correct page (or, for cross page questions, relevant chunks frommorethanonepage)andwerestillmarkedungrounded."

Discussion¶

Related Claims¶

• The topic-page groundedness gap between wiki and vector RAG was the study's only comparison not reaching significance at the 95% level — related
• Standard semantic retrieval alone does not reliably recover the pedagogically required chunk types, with comparison chunks retrieved worst (14.3% on the real corpus) — related
• In a ten-request live textbook audit, citation integrity coexisted with substantive error: every citation resolved to its source, yet answers included retrieval misses and contradictions — related
• The wiki's advantage over vector RAG is roughly three times larger on cross-page questions than on topic-page questions, in both score and groundedness — related
• NLI analysis reveals reference-based grading is limited: many semantically correct responses contradict reference answers — related
• A git-history audit found 96.3% of AI-generated answer-key question blocks needed no post-release semantic repair — related
• Retrieval-augmented generation reduces but does not eliminate hallucinations in AI-assisted verification — related
• Generated answers sometimes added correct background knowledge not explicitly supported by retrieved evidence, an unsupported-answer behavior the authors flag as requiring faithfulness evaluation — related
• Manual error analysis finds incomplete retrieval is the most frequent failure category, ahead of code-semantics limits, bilingual retrieval gaps, and over-conservative refusals — reports the opposite
• Evidence tracing links 96 of 114 generated claims to source chunks, but three claims with retrieval scores of 0.50 or below show retrieval similarity alone is insufficient — related
• Fifth-grade history textbook excerpts largely fail to answer the frame slot questions, presenting detailed action but missing main ideas — related