Multiple Modalities for Perceiving Information¶

Strategy · All strategies
Evidence · 3 claims (1 for, 2 against) · 11 studies (7 causal, 3 quant-synthesis, 1 review), q2–q4 · 3 of 11 report an effect size

Description¶

Presenting information in more than one sensory format — text, audio, video, diagrams, animations, simulations, or interactive models — so learners can access content through the channel best suited to the material, the task, and their abilities. The goal is not decoration or preference-matching but complementary representation: each modality should carry part of the meaning or make the same meaning accessible in a different way.

Design Implications¶

Pairing words with relevant pictures or narration improves learning compared with words alone, because verbal and visual channels are processed separately and can share the load Dual coding improves recall when verbal and visual representations are integrated. [+S]. The benefit is conditional, however: modalities must be complementary and integrated, not redundant or competing. Presenting identical text on screen while narrating it, or adding decorative images, adds load without adding meaning Decorative illustrations do not improve learning. [-S]. Poorly combined modalities can overload working memory and degrade learning Cognitive overload degrades learning when demands exceed working memory capacity. [-S].

Context¶

Requirements¶

• A clear analysis of which content is inherently visual, verbal, spatial, or dynamic, so modality choices follow the content rather than novelty
• Integration of modalities: captions near visuals, narration synchronized with animation, not separated in time or space
• Accessible alternatives for every modality — captions, transcripts, alt text, keyboard-navigable interactives — so multiple modalities function as access routes, not barriers
• Alignment with Cognitive Load Management: each added modality must earn its place by reducing extraneous load or adding representational power

Constraints¶

• Redundant modality pairing (identical on-screen text plus narration) can hurt learning by forcing parallel processing of duplicate verbal input [~S]
• Adding a second modality does not help when the first already suffices; extra channels consume limited working memory Cognitive overload degrades learning when demands exceed working memory capacity. [-S]
• The "learning styles" rationale — matching modality to a learner's preferred style — lacks supporting evidence; modality should match the content, not a presumed style [X]
• Interactive simulations and 3D models impose high extraneous load for novices without guidance or orientation

Implementation Variability¶

• Sequential alternatives: the same content offered as text, podcast, and video, with learners choosing one route (accessibility and flexibility function)
• Complementary combination: narration plus diagram, where each channel carries different information (multimedia principle)
• Interactive representation: simulations and manipulable models where learners generate the visual dynamic themselves
• Temporal adaptation: video for processes that unfold over time; static diagrams for structures that benefit from simultaneous inspection

Target Learners¶

• Learners with sensory disabilities or language-processing differences, for whom alternative modalities are an access requirement rather than an enhancement [+M]
• Novices learning spatially or dynamically complex content (e.g., circulatory systems, mechanical processes), who benefit from animation plus narration [+M]
• Second-language learners, who benefit from captions pairing speech with text [+W]
• Less beneficial — and sometimes harmful — when modality is added merely to match a self-reported preference, or for learners who already have fluent text-based strategies [~W]

Target Learning Goals¶

• Conceptual understanding of systems, structures, and dynamic processes
• Retention of verbally and visually coded information Dual coding improves recall when verbal and visual representations are integrated. [+S]
• Accessibility and equitable participation across abilities

Instructions¶

1. Analyze the content: identify what is inherently verbal, visual, or dynamic, and let that drive modality selection
2. Prepare the core representation in the most load-appropriate channel (e.g., narration for dynamic process, Chunking of text for dense information)
3. Add a complementary representation in a second channel — diagram, animation, or physical model — integrated spatially and temporally with the first
4. Provide accessible equivalents: captions, transcripts, alt text, tactile or audio alternatives
5. Give learners guided interaction with the representation — Practice interpreting, manipulating, or reproducing it — rather than passive viewing
6. Apply and assess: require learners to use the representations in an Application task and check comprehension across modalities

Related Strategies¶

• Activating prior knowledge — preparing learners before a complex multimodal presentation reduces the load of interpreting a new representation
• Accessible vocabulary and syntax — the textual counterpart to modality accessibility; both remove unnecessary processing barriers
• Teach L2 comprehension through a multiliteracies curriculum with video-based visualizing and verbalizing activities
• Integrate mixed-reality visualization with physical interaction rather than relying on a single modality
• Design intuitive, multisensory, layered visualizations with evolving assessment to reduce cognitive load

Examples¶

• PhET Interactive Simulations (https://phet.colorado.edu) — research-validated physics and chemistry simulations combining visual manipulation with optional guided-inquiry activities; widely used for dynamic processes that static text cannot show
• Khan Academy (https://www.khanacademy.org) — narrated video with synchronized visual work, plus transcripts and captions, exemplifying complementary narration-and-visual pairing
• CAST UDL Guidelines (https://udlguidelines.cast.org) — the "multiple means of representation" principle operationalizes this strategy as a universal design requirement, with concrete accessibility checkpoints
• Newsela (https://www.newsela.com) — leveled text alternatives to the same article, illustrating modality/difficulty alternatives for access rather than decoration

Key Sources¶

• Mayer, R. E. (2021). Multimedia Learning (3rd ed.). Cambridge University Press. doi:10.1017/9781316941355
• Paivio, A. (1990). Mental Representations: A Dual Coding Approach. Oxford University Press. doi:10.1093/acprof:oso/9780195066661.001.0001
• Moreno, R., & Mayer, R. E. (2007). Interactive multimodal learning environments. Educational Psychology Review, 19(3), 309–326. doi:10.1007/s10648-007-9047-2
• Pashler, H., McDaniel, M., Rohrer, D., & Bjork, R. (2008). Learning styles: Concepts and evidence. Psychological Science in the Public Interest, 9(3), 105–119. doi:10.1111/j.1539-6053.2009.01038.x
• Clark, R. C., & Mayer, R. E. (2016). E-Learning and the Science of Instruction (4th ed.). Wiley. doi:10.1002/9781119239086