Pedagogy experts prefer LearnLM over contemporaneous flagship models across comparative assessment categories¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r2 · q3
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person · recheck by 2027: tool and model accuracy changes quickly

Subclaims¶

q3 i? Expert assessors showed a strong preference toward LearnLM over GPT-4o for all five comparative assessment categories, with smaller preferences over Claude 3.5 Sonnet and Gemini 1.5 Pro. → LearnLM Team 2024 q3 i? Average preference strength favored LearnLM by +31% over GPT-4o, +11% over Claude 3.5 Sonnet, and +13% over the Gemini 1.5 Pro model it was based on. → LearnLM Team 2024 (2)

Evidence¶

LearnLM Team 2024¶

LearnLM Team, Google. (2024). LearnLM: Improving Gemini for Learning. arXiv. https://arxiv.org/abs/2412.16429 doi:10.48550/arxiv.2412.16429

q3 · i? · design · r2

Stage 3 expert assessment of 2360 collected role-played tutoring conversations, analyzed with hierarchical Bayesian regression. The results section reports that comparative preference ratings "reveal a strong preference toward LearnLM over GPT-4o for all five comparative assessment categories", with similar but smaller preferences over Claude 3.5 Sonnet and Gemini 1.5 Pro.

"First, comparative preference ratings (Figure 4) reveal a strong preference toward LearnLM over GPT-4o for all five comparative assessment categories."

LearnLM Team 2024 (2)¶

LearnLM Team, Google. (2024). LearnLM: Improving Gemini for Learning. arXiv. https://arxiv.org/abs/2412.16429

q3 · i? · causal · r2

The report's summary of the same evaluation prints the aggregate preference strengths: "average preference strength of+31% over GPT-4o,+11% over Claude 3.5 Sonnet, and+13% over the original Gemini 1.5 Pro". These are preference-strength percentages, not standardized effect sizes.

"educators and pedagogy experts preferred this version of LearnLM with an average preference strength of+31% over GPT-4o,+11% over Claude 3.5 Sonnet, and+13% over the original Gemini 1.5 Pro (see Figure 1)."

Discussion¶

Related Claims¶

• Role-playing participants prefer LearnLM over GPT-4o but express no substantial preference versus Gemini 1.5 Pro or Claude 3.5 Sonnet — related
• LearnLM receives the highest expert pedagogy-rubric marks across rubric categories, with the largest leads in active learning, metacognition, and curiosity — related
• Preference explanations favoring LearnLM more often cite staying on topic, challenging the learner, and not giving away answers — related
• Role-playing learners report weaker experiences with GPT-4o than with LearnLM, Gemini 1.5 Pro, and Claude 3.5 Sonnet — related