In an 18-participant user study, the system received a 4.00/5 overall recommendation, with hallucination resistance rated 4.44/5 and highest ratings for ease of starting a conversation (4.39/5) and response relevance and accuracy (4.06/5)¶

Claim · All claims
Evidence · 1 study (2 entries) · design r2 · q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly

Subclaims¶

Reading the evidence codes

q is the design tier, i the impact, n the sample, and the kind with its r the rigour judged within that kind: what each code means.

This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.

q2 i? Participants rated resistance to hallucinations high (Mean = 4.44/5), confirming RAG was effectively limited to teacher-provided sources. → Anna Ostrowska 2026 q2 i? The overall recommendation score was 4.00 out of 5, with ease of starting a conversation (4.39/5) and relevance and accuracy of responses (4.06/5) rated highest; 14 of 18 users named the chat interface and automatic test generation as most useful. → Anna Ostrowska 2026 (2)

Evidence¶

Anna Ostrowska 2026¶

Anna Ostrowska, Michał Kukla, Gabriela Majstrak, Jan Opala, Sebastian Pergała, Jan Skwarek, Anna Wróblewska. (2026). From Surface Learning to Deep Understanding: A Grounded AI Tutoring System for Moodle. https://tinyurl.com/mrytc2dr doi:10.24963/ijcai.2026/984

q2 · i? · design · r2

Human feedback study with 18 participants (83.3% students, 11.1% instructors) using a Likert scale (1–5) and open-ended questions, interacting with the chatbot, quizzes, and dashboard. "Participants consistently rated the resistance to hallucinations as high (Mean = 4.44/5)".

"Participants consistently rated the resistance to hallucinations as high (Mean = 4.44/5), confirming that RAG was effectively limited to teacher-provided sources"

Anna Ostrowska 2026 (2)¶

q2 · i? · design · r2

The same 18-participant Likert-scale usability study reports "The overall recommendation score for the system was 4.00 out of 5", with the printed subscale means for conversation ease and response relevance and accuracy.

"The overall recommendation score for the system was 4.00 out of 5, with the highest individual ratings given to the ease of starting a conversation (4.39/5) and the relevance and accuracy of responses (4.06/5)"

Discussion¶

Page metadata

Field	Value
`type`	`claim`
`id`	`user-study-18-participants-recommendation-4-of-5`
`title`	`In an 18-participant user study, the system received a 4.00/5 overall recommendation, with hallucination resistance rated 4.44/5 and highest ratings for ease of starting a conversation (4.39/5) and response relevance and accuracy (4.06/5)`
`description`	`In an 18-participant user study, the system received a 4.00/5 overall recommendation, with hallucination resistance rated 4.44/5 and highest ratings for ease of starting a conversation (4.39/5) and response relevance...`
`status`	`review`
`evidence_strength`	`moderate`
`newest_evidence`	`2026`
`recheck_by`	`2029`
`generated`	by `process:wiki-ingest` on `2026-10-11`
`trust`	unverified — no one has confirmed this page's content
`sources`	2 — `anna-ostrowska-2026`, `anna-ostrowska-2026-2`

Frontmatter is the page's machine-readable record — see the Schema & Guide.