Users rate chatbots easy to use with clear, relevant, largely accurate responses, but the intrusive pop-up design is the largest annoyance¶

Claim · All claims
Evidence · 1 study (2 entries) · associational r1 · q3
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Users rated the chatbots easy to use, clear, relevant and accurate, but reported occasional errors, and the intrusive automatic pop-up was the biggest usability complaint among both users and non-users. → Stephen Colbran 2026 q3 i? Non-users' largest open-ended theme was usability (23% of responses), centred on disliking the intrusive pop-up design. → Stephen Colbran 2026 (2)

Evidence¶

Stephen Colbran 2026¶

Stephen Colbran, Meena Jha and Caitlin Schiavone. (2026). Understanding student perspectives on generative AI chatbots: a human-centred mixed-methods study in higher education. International Journal of Educational Technology in Higher Education. https://doi.org/10.1186/s41239-026-00605-w

q3 · i? · associational · r1

One-sample t-tests in the RQ4 results (user subgroup): users found the chatbot "easy to use r ( p < .001, n = 37)", responses clear and relevant, information accurate (p < .001, n = 38), personalised (p = .026), and reported occasional errors (p = .003).

"Users thought that the chatbots were easy to use r ( p < .001, n = 37), that the chatbot responses were clear and understandable ( p < .001, n = 37), that chatbot responses were relevant to their questions or needs ( p < .001, n = 38)."

Stephen Colbran 2026 (2)¶

Stephen Colbran, Meena Jha and Caitlin Schiavone. (2026). Understanding student perspectives on generative AI chatbots: a human-centred mixed-methods study in higher education. International Journal of Educational Technology in Higher Education. https://doi.org/10.1186/s41239-026-00605-w

q3 · i? · qualitative · r3

Reflexive thematic analysis of non-user open-ended comments (Fig. 5): "Usability was the biggest theme (23% of responses) with many students disliking the intrusive pop-up design"; example comments described the pop-up as annoying and intrusive. Among improvement suggestions, pop-up and user-experience issues accounted for 22% of responses.

"Usability was the biggest theme (23% of responses) with many students disliking the intrusive pop-up design."

Discussion¶

Related Claims¶

• Usability testing with educators and developers identified chatbot strengths for independent challenge-solving but limited feedback for low-participation learners — related
• In a pilot study, the TigerGPT survey chatbot achieved a 75% usability rating and an 81% satisfaction score, with 50% of participants preferring it over traditional surveys — related
• Non-users cite lack of training, distrust of responses, preference for teaching staff, academic-integrity fear and lack of interest as reasons for not using chatbots — related
• In an 18-participant user study, the system received a 4.00/5 overall recommendation, with hallucination resistance rated 4.44/5 and highest ratings for ease of starting a conversation (4.39/5) and response relevance and accuracy (4.06/5) — related
• The FACTRIA-aware chatbot received good usability ratings (SUS 70.84) and high usefulness ratings from most of the 11 stakeholders — related
• Participants rated SOPHIE 2.0 as engaging, easy to use, and clinically relevant in a self-service format — related
• 91% of UAT participants rated the Taklif.AI user interface as intuitive — related
• General-purpose LLMs show accuracy and reliability limits, including hallucinations and task-specific accuracy below perfect, making them risky as expert advice sources — related