In deployment, 13% of sampled LLM agent responses were faulty and 37% very useful, with response quality improving over weeks of use¶
Claim · All claims
Evidence · 1 study · causalr2·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Human evaluation of 284 randomly sampled LLM responses found 13% faulty and 37% likely very useful, with faulty proportions decreasing in later weeks. → Graf 2026
Evidence¶
Graf 2026¶
Graf, L., Bassner, P., Anzinger, M., Dietrich, F., Krusche, S., & Poquet, O. (2026). Interactive learning dashboards: rethinking learning visualisations as engagement tools. Education and Information Technologies. https://doi.org/10.1007/s10639-026-14082-1
q2 · i? · causal · r2
System evaluation of agent response sensibility: 284 randomly sampled responses rated by humans. In the first two weeks 37% were faulty and only 27% very useful, indicating students learned what the chatbot was useful for.
"Overall, 13% of the randomly sampled LLM responses were labelled as faulty (not fully matching the request by human common sense or containing false or misleading claims). 37% of them were rated as likely very use-ful for the student."
Discussion¶
Related Claims¶
- Qualitative analysis found staged disclosure sometimes withheld inaccurate diagnostic details, but some errors still shaped student revisions — related
- An agentic AI instructor sustained and deepened student engagement across successive course modules in a field deployment — related
- A three-week MALA deployment recorded 62 distinct usernames across 128 conversations, with 63% of conversations resolved or partially resolved — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
llm-agent-response-quality-evaluation |
title |
In deployment, 13% of sampled LLM agent responses were faulty and 37% very useful, with response quality improving over weeks of use |
description |
In deployment, 13% of sampled LLM agent responses were faulty and 37% very useful, with response quality improving over weeks of use |
status |
review |
evidence_strength |
weak |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-11 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — graf-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.