Students using Google produce higher-quality justifications in their final recommendations than students using ChatGPT 3.5¶

Claim · All claims
Evidence · 1 study · causal r2 · q3 · i2 medium
Currency · newest study 2024 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person

Subclaims¶

q3 i2 Students in the web search condition listed significantly more relevant arguments in their final recommendations than students in the LLM condition, adjusted for prior knowledge (η² = 0.11). → Matthias Stadler 2024

Evidence¶

Matthias Stadler 2024¶

Matthias Stadler, Maria Bannert, Michael Sailer. (2024). Cognitive ease at a cost: LLMs reduce mental effort but compromise depth in student scientific inquiry. Computers in Human Behavior. https://doi.org/10.1016/j.chb.2024.108386

q3 · i2 · causal · r2

ANCOVA on coded justification quality (number of relevant aspects mentioned; inter-rater agreement k = 0.92) in the randomized experiment (n=91). Web search mean 1.87 (SD 1.10) versus LLM 1.20 (SD 0.77), with "11.18 0.001 0.11", i.e., η² = 0.11, a medium effect.

"H2 –Qualityofjustifications 1.87(1.10) 1.20(0.77) 11.18 0.001 0.11"

Studies bearing on this claim¶

When learners research a question, does searching with a web search engine lead to deeper learning than asking a generative AI chatbot or reading an AI summary? This table gathers the studies behind this claim and the 6 narrower claims that bear on it: 3 distinct studies (3 causal), q2–q3, 1 of 3 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Related Claims¶

• Direct GenAI-supported feedback improves immediate argument-quality gain over peer feedback, while reflective and hybrid designs outperform direct GenAI feedback — the general claim this finding bears on, with the studies behind it
• Recommendation homogeneity does not differ between students using ChatGPT 3.5 and students using Google — related
• Students using ChatGPT 3.5 experience lower extraneous cognitive load than students using Google during a web research task — related
• Germane cognitive load fully mediates the effect of information-search tool on justification quality (exploratory analysis) — related
• ChatGPT users differentiated information less (never by credibility) and rarely verified information compared to Google users — related
• Cognitive offloading to generative AI is associated with reduced learner engagement, lower critical thinking, and lower-quality reasoning — a broader claim this one bears on
• Google Trends data show ChatGPT search interest surged globally within weeks of its November 2022 public release, with plagiarism among the top related queries — related
• Undergraduate students’ genAI knowledge contains specific confident misconceptions, most prevalently that AI chatbots perform web searches — related
• ChatGPT users showed a non-significant trend toward worse overall informal learning outcomes than Google Search users over 8 days — related
• The impact of the strategy training was higher for evaluation than for search skills, in terms of accuracy and quality of students' justifications. — related
• ChatGPT biased information access toward solution-oriented artifacts over principle-oriented information — related
• LLM-supported socio-scientific inquiry lowered cognitive load but produced lower-quality reasoning and argumentation than conventional search — a broader claim this one bears on