Frame LLM integration as an assessable AI literacy competency with differentiated integrity policies and reasoning-focused assessment redesign¶

Strategy · All strategies
Evidence · 4 claims (4 for) · 1 study (1 review), q3 · 1 of 1 report an effect size · 4 claims rest on one study

Description¶

The review recommends a three-part practice response to its synthesis: treat LLM use as "a learnable competency rather than a prohibited tool or an unregulated resource", incorporate AI literacy—"the ability to critically appraise, verify, and appropriately weight LLM-generated content"—as an explicit assessable outcome, replace recall-heavy multiple-choice with formats evaluating clinical reasoning and uncertainty management, and adopt institution-specific integrity policies distinguishing constructive from dishonest use, since blanket prohibition is likely ineffective.

Design Implications¶

Context¶

Requirements¶

• Formal incorporation of AI literacy as an explicit and assessable learning outcome within competency-based medical education frameworks.

Constraints¶

• Blanket prohibition policies without enforcement capacity risk institutional credibility deficits; assessments relying predominantly on factual recall are particularly vulnerable to LLM completion.

Target Learners¶

• Undergraduate medical students (preclinical and clinical years)
• Medical school faculty

Target Learning Goals¶

• AI literacy (critical appraisal, verification, and appropriate weighting of LLM-generated content)
• Clinical reasoning and applied judgement

Claims¶

• GPT-4 exceeds pass marks on India NEET-PG and UK PLAB 1 but falls below pass on clinical reasoning and procedural items [+W]
• GPT-4 examination accuracy is moderated by item type: multi-step vignettes (61–71%) and image-dependent items (48–63%) fall well below single-fact recall (83–88%) [+W]
• LLM use as a supplementary learning aid yields modest short-term knowledge gains (d = 0.31–0.71) that attenuate without durable retention [+W]
• GPT-4 achieves passing accuracy on USMLE Steps 1–3 (mean 86%), significantly exceeding GPT-3.5 (mean 56%) [+M]

Related Strategies¶

• Renew writing assessment and integrity practice through AI-literacy instruction, process-focused assessment design, and contextual detector training
• Institutional recommendations: shift from detection to process-oriented assessment, transparent AI-use protocols, and revised integrity policies
• Institutional governance should establish practical conditions supporting AI literacy, learner agency, and critical evaluation rather than relying on prohibition and detection
• Integrate GenAI into HSS education through scaffolded, practice-oriented training sequences and discipline-responsive, process-focused assessment
• Fund AI literacy initiatives integrating AI ethics into curricula and adapting assessments
• Redesign assessment toward progressive, authentic, and reflective methods rather than intensive paperwork
• Strategies for integrating GenAI in STEAM teaching: outcome-anchored lesson planning, differentiation, inquiry and project variants, assessment redesign, and integrity as pedagogy
• Adapt non-invigilated written assessments and their marking criteria in response to Gen-AI susceptibility
• Adopt a differentiated integration strategy for GenAI in undergraduate medical education rather than uniform adoption

Examples¶

-

Key Sources¶

• Bodke Parag, Jumle Rahul. (2026). Large language models in undergraduate medical education: a systematic review of examination performance, knowledge acquisition, and learner perceptions. BMC Medical Education. https://doi.org/10.1186/s12909-026-09999-7