Non-expert participants often fail to notice empathy differences between chatbot versions, so self-reported perceived empathy alone is an unreliable assessment method¶
Claim · All claims
Evidence · 1 study · qualitativer?·q2
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person · recheck by 2029: tool and model accuracy changes quickly
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? Many participants did not notice the empathy differences between chatbot versions, supporting the article's argument that relying solely on non-expert perceived empathy ratings is not a reliable assessment method. → Li Siyan 2026
Evidence¶
Li Siyan 2026¶
Li Siyan, Kai-Hui Liang, Shopnil Shahriar, Yilin Ye, Shiyoh Goetsu, Wei-Wei Du, Masahiro Yoshida, Tsunayuki Ohwa, Xuhai Xu, and Zhou Yu. (2026). Invisible Impact of Empathy on Behavioral Change: Isolating the Effect of Empathy in Long-term Physical Activity Coaching Chatbot Interactions. https://arxiv.org/abs/2606.26641 doi:10.48550/arxiv.2606.26641
q2 · i? · qualitative · r?
Exit interviews with nine participants in the longitudinal user study, focusing on overall experience and whether users could detect tone shifts between weekly chatbot versions. The article reports that "many participants did not notice the empathy differences between chatbot versions"; no effect size is printed.
"One interesting phenomenon is thatmany participants did not notice the empathy differences between chatbot versions."
Discussion¶
Related Claims¶
- Average check-in and weekly ratings show no significant differences between chatbot conditions, with the Non-Empathetic chatbot often rated as more engaging and useful — related
- LLM-annotated empathy dimensions correlate positively with intention to follow, self-efficacy, and chatbot usefulness despite participants not preferring empathetic chatbots — related
- In a six-week within-subject study, the Empathetic PA coaching chatbot improves intention to follow and self-efficacy faster over time than the Non-Empathetic condition — related
- LLM-annotated EPITOME scores show the Non-Empathetic chatbot expresses significantly lower empathy on all three dimensions, while Standard and Empathetic chatbots are not significantly different — related
- Starting interaction with the Empathetic chatbot is associated with the least attrition in the six-week study — related
- Self-actualization (POI) correlates of rated counseling performance reported by Foulds could not be confirmed on replication — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
nonexpert-perceived-empathy-unreliable |
title |
Non-expert participants often fail to notice empathy differences between chatbot versions, so self-reported perceived empathy alone is an unreliable assessment method |
description |
Non-expert participants often fail to notice empathy differences between chatbot versions, so self-reported perceived empathy alone is an unreliable assessment method |
status |
review |
evidence_strength |
moderate |
newest_evidence |
2026 |
recheck_by |
2029 |
generated |
by process:wiki-ingest on 2026-10-10 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — li-siyan-2026 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.