Students using ChatGPT 3.5 experience lower intrinsic cognitive load than students using Google during a web research task¶

Claim · All claims
Evidence · 1 study · causal r2 · q3 · i3 large
Currency · newest study 2024 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i3 Students in the LLM condition reported significantly lower intrinsic cognitive load than students in the web search condition, adjusted for prior knowledge (η² = 0.15). → Matthias Stadler 2024

Evidence¶

Matthias Stadler 2024¶

Matthias Stadler, Maria Bannert, Michael Sailer. (2024). Cognitive ease at a cost: LLMs reduce mental effort but compromise depth in student scientific inquiry. Computers in Human Behavior. https://doi.org/10.1016/j.chb.2024.108386

q3 · i3 · causal · r2

ANCOVA in the same randomized experiment (n=91) controlling for prior knowledge. LLM group mean ICL was 3.16 (SD 1.46) versus 4.43 (SD 1.60) for web search, with "16.48 <0.001 0.15", i.e., η² = 0.15, a large effect.

"H1.2-ICL 4.43(1.60) 3.16(1.46) 16.48 <0.001 0.15"

Discussion¶

Learner Variables¶

• Working Memory — outcome: instruction changes it

Related Claims¶

• Students using Google produce higher-quality justifications in their final recommendations than students using ChatGPT 3.5 — the general claim this finding bears on, with the studies behind it
• Students using ChatGPT 3.5 experience lower extraneous cognitive load than students using Google during a web research task — related
• Students using ChatGPT 3.5 experience lower germane cognitive load than students using Google, the strongest of the three load differences — related
• High- and low-prior-knowledge learners differ significantly on an algebra posttest and on intrinsic and germane cognitive load, but not on extraneous load — related
• Recommendation homogeneity does not differ between students using ChatGPT 3.5 and students using Google — related
• Germane cognitive load fully mediates the effect of information-search tool on justification quality (exploratory analysis) — related
• Pragmatic and contingent LLM rewrites reduced perceived cognitive load relative to standard interpreter messages — related
• ChatGPT participants experienced meta-cognitive load and frictions, with slightly more unsatisfied endings than Google participants — related
• LLM-supported socio-scientific inquiry lowered cognitive load but produced lower-quality reasoning and argumentation than conventional search — a broader claim this one bears on
• Tracing and pointing does not significantly reduce self-reported intrinsic cognitive load — related