Combine quantitative RAGAS metrics with human rubric-based claim auditing to detect hallucination and weak grounding in LLM-assisted forum analysis¶

Strategy · All strategies
Evidence · no claims cited

Description¶

The article recommends evaluating LLM-generated analytical responses with both automated metrics and structured human review, because "automated scores alone may not fully capture weak grounding, retrieval mismatch, or unsupported interpretation." The human audit decomposes each response into claims, traces each claim to its source chunk and original Reddit record, and rates evidence presence, support strength (strong, partial, weak), and interpretative utility. This hybrid approach surfaced hallucination risk that RAGAS scores alone missed.

Design Implications¶

Context¶

Requirements¶

• Responses must carry claim-level source-chunk identifiers so each claim can be traced to original records.
• A rubric-based guideline covering hallucination risk, evidence alignment, and interpretability must be developed for assessors.

Constraints¶

• The audit was conducted on a purposive sample of 100 claims selected to include both strong and weak RAGAS score patterns, not on all generated claims.

Target Learners¶

• Researchers analysing low-resource online forum discourse

Target Learning Goals¶

• Assessing verifiability and hallucination risk in LLM-generated research responses

Related Strategies¶

• Design guidelines for future LLM evaluation tools: configurable metric views, tighter metric-rubric integration, and comparison-foregrounding visualizations

Examples¶

-

Key Sources¶

• Dana Rezazadegan, Atie Kia, Phongpadid Nandavong, Dominique Carlon, Jeremy Nguyen, Abhik Banerjee, James Marshall, Anthony McCosker, Yong-Bin Kang. (2026). DysLexLens: A Low-Resource LLM Framework for Analysing Dyslexic Learners' Insights from Online Forums. https://arxiv.org/abs/2606.27619