Role-playing learners report weaker experiences with GPT-4o than with LearnLM, Gemini 1.5 Pro, and Claude 3.5 Sonnet¶

Claim · All claims
Evidence · 1 study · design r2 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i? Participants role-playing learners reported relatively similar experiences with LearnLM, Gemini 1.5 Pro, and Claude 3.5 Sonnet, but weaker experiences with GPT-4o in interest, perceived warmth, and perceived usefulness. → LearnLM Team 2024

Evidence¶

LearnLM Team 2024¶

LearnLM Team, Google. (2024). LearnLM: Improving Gemini for Learning. arXiv. https://arxiv.org/abs/2412.16429 doi:10.48550/arxiv.2412.16429

q3 · i? · design · r2

Stage 2 survey data from participants who role-played learners (Figure 6), covering interest in the topic, willingness to use the tutor in the future, and perceived competence and warmth. The text reports participants "indicatedweaker experiences with GPT-4o" on interest, warmth, and usefulness, while experiences with the other three systems were relatively similar.

"In contrast, participants indicatedweaker experiences with GPT-4o in terms of its effects on their interest, its perceived warmth, and its perceived usefulness."

Discussion¶

Related Claims¶

• Preference explanations favoring LearnLM more often cite staying on topic, challenging the learner, and not giving away answers — related
• Role-playing participants prefer LearnLM over GPT-4o but express no substantial preference versus Gemini 1.5 Pro or Claude 3.5 Sonnet — possibly the same claim (merge candidate)
• Pedagogy experts prefer LearnLM over contemporaneous flagship models across comparative assessment categories — related
• Perceived warmth and competence of an AI tutor positively predict learners' willingness to use it in the future — related