LLM use as a supplementary learning aid yields modest short-term knowledge gains (d = 0.31–0.71) that attenuate without durable retention¶

Claim · All claims
Evidence · 1 study (2 entries) · review r3 · q3 · i1 small
Currency · newest study 2026 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q3 i1 Across six studies, standardised mean differences for LLM-assisted learning ranged from d = 0.31 to d = 0.71 (median ~0.53), largest in biochemistry, pathology and pharmacology. → Bodke Parag 2026 q3 i? Retention at 4–8 weeks was markedly attenuated, with two of four studies showing no statistically significant retention advantage (p = 0.12 and p = 0.09). → Bodke Parag 2026 (2)

Evidence¶

Bodke Parag 2026¶

Bodke Parag, Jumle Rahul. (2026). Large language models in undergraduate medical education: a systematic review of examination performance, knowledge acquisition, and learner perceptions. BMC Medical Education. https://doi.org/10.1186/s12909-026-09999-7

q3 · i1 · review · r3

Narrative synthesis of 12 pre/post or validated-instrument knowledge acquisition studies; a pooled estimate was not calculated due to heterogeneity. Printed effect sizes span d = 0.31 to 0.71, "median approximately 0.53".

"Across the six studies reporting quantifiable effect sizes, standardised mean differences ranged from d = 0.31 to d = 0.71 (median approximately 0.53)"

Bodke Parag 2026 (2)¶

Bodke Parag, Jumle Rahul. (2026). Large language models in undergraduate medical education: a systematic review of examination performance, knowledge acquisition, and learner perceptions. BMC Medical Education. https://doi.org/10.1186/s12909-026-09999-7

q3 · i? · review · r3

Retention follow-ups at 4–8 weeks reported in the knowledge acquisition section; two of four studies showed no statistically significant retention advantage, with printed p-values 0.12 and 0.09. No effect size printed.

"Retention assessments conducted at 4–8 weeks post-intervention showed marked attenuation, with two of four studies reporting no statistically significant knowledge retention advantage in the LLM-assisted group (p = 0.12 and p = 0.09, respectively)"

Discussion¶

Related Claims¶

• Evidence on GenAI in chemistry education is concentrated in short-term outcomes, with durable transfer and retention rarely tested — a broader claim this one bears on
• Product-outcome studies report GenAI-related gains on product-quality measures but do not assess durable, transferable learning — a broader claim this one bears on
• Falsifiable hypothesis: unrestricted AI use on deep-processing tasks will produce a product–process dissociation—higher-rated assignments but lower unaided delayed transfer — related
• AI-assisted task performance can rise without corresponding gains in underlying knowledge, metacognitive processing, or independent reasoning — related
• The decisive empirical test of AI-augmented instruction frameworks is whether students retain representational competence when AI support is withdrawn — related
• The article reports observational patterns that AI assistance can improve short-term performance while reducing cognitive engagement and independent capability — related