Multisensory Encoding¶

Strategy · All strategies
Evidence · 7 claims (2 for, 1 mixed, 4 against) · 18 studies (8 causal, 5 quant-synthesis, 4 review, 1 theoretical), q2–q4 · 5 of 18 report an effect size · 2 claims rest on one study

Description¶

Multisensory encoding presents the same core content through more than one sensory modality — most commonly combining visual and auditory channels, but also including gesture, movement, and tactile experience. The goal is not decoration or redundancy but complementary representations: each channel carries part of the load or reinforces the same structure in a different format.

Design Implications¶

Multisensory presentation works because separate processing channels for visual and auditory input allow parallel information intake, and because dual-format traces improve retrieval Dual coding improves recall. [+S]. The effect is strongest when each modality adds information rather than duplicating it verbatim: narrated animation outperforms on-screen text read aloud, which overloads the visual channel Multimedia learning is improved by presenting words as narration rather than on-screen text. [+S]. Adding irrelevant sensory material — background music, decorative images — impairs rather than helps learning Decorative illustrations do not improve learning. [-S].

Context¶

Requirements¶

• Content that can be genuinely split across channels (e.g., narration describing what a diagram shows), not merely duplicated
• Alignment between modalities: visuals and words must refer to the same content at the same time (Contiguity)
• Modality choices that respect total working memory capacity, since each added channel can also add load Cognitive overload degrades learning. [-S]

Constraints¶

• Redundant presentation — identical text spoken and displayed — hurts learning compared with narration alone Redundant on-screen text with narration impairs learning. [-M]
• Splitting attention between two visual sources (diagram plus text describing it) forces costly integration Split attention between sources impairs learning. [-M]
• Benefits shrink for learners with strong prior knowledge, who can integrate representations on their own Worked-example guidance becomes less effective as learner expertise increases. [~M]
• For learners with sensory processing differences, added modalities can distract rather than support; modality choice must be adjustable

Implementation Variability¶

• Audio-visual: narration synchronized with animation or diagrams (the canonical multimedia case)
• Enactive: gesture, manipulatives, or movement tied to concepts (Act It Out)
• Tactile: physical models and manipulatives, especially in early mathematics and science
• Text-plus-visual: annotated diagrams and Annotating as a learner-generated dual-code

Target Learners¶

• Novices, who benefit most from complementary representations that reduce integration demands Dual coding improves recall. [+S]
• Beginning readers and language learners, for whom pictures scaffold word meaning
• Less beneficial for experts, for whom extra representations can be redundant Worked-example guidance becomes less effective as learner expertise increases. [~M]

Target Learning Goals¶

• Recall and comprehension of verbal material paired with imagery Dual coding improves recall. [+S]
• Mental model construction for dynamic systems and processes
• Vocabulary and concept acquisition in early literacy and second-language learning

Instructions¶

1. Identify the core content and decide which channel should carry it — narration for verbal explanation, visuals for spatial or dynamic structure (Chunking the content first helps)
2. Create or select visuals that are essential, not decorative Decorative illustrations do not improve learning. [-S]
3. Synchronize narration with the corresponding visual segment; avoid displaying the narration as on-screen text Multimedia learning is improved by presenting words as narration rather than on-screen text. [+S]
4. Add a learner activity that requires integrating the modalities, such as labeling a diagram or explaining the visual in words (Annotating)
5. Check total load; remove any channel that duplicates rather than complements Redundant on-screen text with narration impairs learning. [-M]

Related Strategies¶

• Dual Coding — the theoretical account of why verbal-plus-visual encoding improves retrieval
• Multimedia Learning — the design principles governing audio-visual combinations
• Chunking — manages the per-channel load that multisensory presentation adds
• Teach L2 comprehension through a multiliteracies curriculum with video-based visualizing and verbalizing activities
• Design intuitive, multisensory, layered visualizations with evolving assessment to reduce cognitive load

Examples¶

• Khan Academy — narrated, hand-drawn video explanations pair spoken reasoning with evolving visuals, followed by practice.
• PhET Interactive Simulations (University of Colorado Boulder) — science simulations combining dynamic visuals with optional narration and embedded feedback.
• Jolly Phonics — early literacy curriculum pairing each phoneme with an action, a song, and a letter shape, encoding sounds enactively and visually.

Key Sources¶

• Paivio, A. (1990). Mental representations: A dual coding approach. Oxford University Press. doi:10.1093/acprof:oso/9780195066661.001.0001
• Mayer, R. E. (2021). Multimedia learning (3rd ed.). Cambridge University Press. doi:10.1017/9781316941355
• Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43–52. doi:10.1207/S15326985EP3801_6
• Ginns, P. (2006). Integrating information: A meta-analysis of the spatial contiguity and temporal contiguity effects. Learning and Instruction, 16(6), 511–525. doi:10.1016/j.learninstruc.2006.10.001
• Shams, L., & Seitz, A. R. (2008). Benefits of multisensory learning. Trends in Cognitive Sciences, 12(11), 411–417. doi:10.1016/j.tics.2008.07.006