On the external CS50-Forum set, EduGuard remains strongest with 86.7% correctness and 6.8% hallucination, though all systems lose grounding¶

Claim · All claims
Evidence · 1 study · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? On the 150-query public course-forum validation set, EduGuard keeps the top ranking with 86.7% correctness and 6.8% hallucination. → S M Asif Hossain 2026

Evidence¶

S M Asif Hossain 2026¶

S M Asif Hossain, Ruksat Khan Shayoni, M. F. Mridha, and Jungpil Shin. (2026). EduGuard: A Safe RAG-Based LLM Tutor for Programming Education. arXiv preprint arXiv:2607.15738. https://arxiv.org/abs/2607.15738 doi:10.48550/arxiv.2607.15738

q2 · i? · design · r2

External validation on the 150-query CS50-Forum set (Table 3); the article describes it as "a public course-forum generalization test", with wider confidence intervals because the set is smaller. Leakage for EduGuard was 8.9% in Table 3.

"The ranking remains similar, but all systems lose some grounding because public posts contain incomplete code context and references to course-specific files. EduGuard remains strongest, with 86.7% correctness and 6.8% hallucination."

Discussion¶

Related Claims¶

• The verification threshold trades hallucination reduction against over-refusal, with the default tau of 0.20 providing the best balance — related
• Manual error analysis finds incomplete retrieval is the most frequent failure category, ahead of code-semantics limits, bilingual retrieval gaps, and over-conservative refusals — related
• DeepSeek-MoE (Ours) achieves the strongest overall balance of accuracy among tested configurations, reaching 78.6% overall and 81.2% multi-hop reasoning accuracy on a 300-query OER evaluation set — related