State-of-the-art conversational agents (Blender and GPT-3) replied adequately to students but were well behind human performance in helping the student (reported from a dialogue study)¶
Claim · All claims
Evidence · 1 study · theoreticalr2·q2
Currency · newest study 2023 · not yet checked against its sources · not reviewed by a person · recheck by 2026: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Blender and GPT-3 replied adequately and conveyed an impression of understanding the learner, but were well behind human performance in helping the student. → Kasneci 2023
Evidence¶
Kasneci 2023¶
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., Stadler, M., Weller, J., Kuhn, J., & Kasneci, G. (2023). ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education. https://doi.org/10.1016/j.lindif.2023.102274
q2 · i? · theoretical · r2
Narrative review attribution in Section 2.2: the paper reports a study of state-of-the-art conversational agents in educational dialogue, noting both Blender and GPT-3 replied adequately yet were "well behind human performance when it comes to helping the student"; no effect size is printed.
"They are however well behind human performance when it comes to helping the student [47], thus emphasizing the need for further research."
Discussion¶
Related Claims¶
- A fine-tuned student model paired with GPT-4 produces conversational statistics closer to human-AI dialogues than two prompt-engineered base models — related
- AI assessors show emerging reliability: acceptable AI-human rater agreement (Cohen's 0.78) in a 12-educator case study and human-comparable feedback helpfulness ratings with 45 learners — related
- A deployed reflection agent followed its prompt's countable rules but broke its dispositional rules about manner — related
- Expert former math teachers rated GPT-4o's dialogue annotations very highly for student correctness and moderate-to-high for knowledge components, with volatile inter-rater reliability. — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
conversational-agents-behind-human-helping-students |
title |
State-of-the-art conversational agents (Blender and GPT-3) replied adequately to students but were well behind human performance in helping the student (reported from a dialogue study) |
description |
State-of-the-art conversational agents (Blender and GPT-3) replied adequately to students but were well behind human performance in helping the student (reported from a dialogue study) |
status |
review |
evidence_strength |
weak |
newest_evidence |
2023 |
recheck_by |
2026 |
generated |
by process:wiki-ingest on 2026-10-15 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — kasneci-2023 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.