Indexing LLM-generated artifacts nearly doubles retrieval recall on analytical queries compared with transcript-only retrieval, while direct queries perform comparably across configurations¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

q2 i? On 20 analytical queries whose evaluative vocabulary does not appear in student speech, transcript-only retrieval achieved Recall@5 = .371, while combining all artifact types yielded the strongest performance (Recall@5 = .739, Recall@10 = .804, MRR@5 = .942). → Dawei Xie 2026 q2 i? On 10 direct queries with terms appearing in transcript text, all four artifact configurations performed comparably (Recall@5 .867–.933; MRR@5 1.000). → Dawei Xie 2026 (2)

Evidence¶

Dawei Xie 2026¶

Dawei Xie, Khalil Anderson, Tochukwu Eze, Chenghong Lin, Bookyung Shin, and Marcelo Worsley. (2026). CLARA: An AI-Augmented Analytics Dashboard for Collaboration Literacy. https://arxiv.org/abs/2605.17259 doi:10.48550/arxiv.2605.17259

q2 · i? · design · r2

Retrieval evaluation of four artifact configurations (transcript-only, +concept maps, +7C assessments, all artifacts) over 30 curated queries with manually identified ground truth, using reciprocal rank fusion. Transcript-only Recall@5 was .371 on analytical queries; adding concept maps gave .534 and 7C assessments .563.

"Combiningallartifacttypesyieldedthestrongestperformance (Recall@5=.739, Recall@10=.804, MRR@5=.942)."

Dawei Xie 2026 (2)¶

Dawei Xie, Khalil Anderson, Tochukwu Eze, Chenghong Lin, Bookyung Shin, and Marcelo Worsley. (2026). CLARA: An AI-Augmented Analytics Dashboard for Collaboration Literacy. https://arxiv.org/abs/2605.17259

q2 · i? · design · r2

For the 10 direct queries whose terms appear in transcripts, retrieval found relevant sessions regardless of which collections were available, so the artifact advantage was specific to analytical queries framed in evaluative vocabulary.

"On direct queries, all configurations performed comparably(Recall@5: .867–.933; Recall@10: .933–.967; MRR@5: 1.000 across conditions)."

Discussion¶

Related Claims¶

• Artifact-grounded agent responses are rated significantly higher than transcript-only responses on groundedness, analytical depth, helpfulness, relevance, and overall quality — related
• A hybrid human-AI workflow using GPT-4o with retrieval-augmented generation supported efficient inductive thematic analysis while preserving researcher judgment — related
• General-purpose embedding yields better retrieval while domain-specific embedding yields higher answer completeness and relevance — related
• The Dashboard's largest rating advantage was on exploring response details (L1, Δ=1.30), while the two platforms were nearly comparable on broad trends (L2, Δ=0.20) and development over time (L6, Δ=0.10) — related
• Retrieval augmentation is essential for on-premise educational QA: a local LLM without retrieval scores below even a TF-IDF baseline — a broader claim this one bears on
• NCERT-tailored RAG preprocessing and figure-aware indexing improve retrieval precision over a generic-chunking baseline, per the authors' assertion — related