Skip to content

Multimodal Instruction

Principle · All principles
Evidence · 42 claims (20 for, 18 mixed, 4 against) · 42 studies (20 causal, 10 review, 7 quant-synthesis, 2 associational, 1 qualitative, 1 design, 1 theoretical), q2–q4 · 10 of 42 report an effect size · 36 claims rest on one study

Conditional relationship

Some content has a structure that one mode shows poorly: a spatial relation, a process unfolding over time, a quantity and its graph, a physical action. For a learner who cannot yet produce or explain that structure, two things are expected to improve comprehension and conceptual understanding over a single mode: presenting the content through two or more coordinated modes, each carrying a part of the meaning the others do not, and asking the learner to translate between them (sketch what an equation says, say what a simulation shows). The modes might be text, speech, a diagram, a simulation, a demonstration, gesture or sign. The outcome is a task that needs the relation, measured soon after instruction (immediate-performance). Whether the gain lasts (delayed-retention) or carries to a new task (near-transfer) is a separate claim, and nothing here tests it.

The relationship is conditional on four things a page cannot settle in advance. The first is the learner's representational competence: whether they can already read each representation and connect it to the content. The second is pacing: who controls the flow, the system or the learner. The third is redundancy: whether two modes carry the same words and compete for the same channel. The fourth is access: whether the learner can hear, see or read the language of each mode. It is a relationship about the content and the task, not about a learner's style. Matching modes to a self-reported "visual" or "auditory" learner type is a different claim, and the evidence is against it (below).

Multimedia Learning holds the narrower conditional model: adding a relevant picture to words, where to place it, decorative images, and segmenting. This page does not repeat it. It owns the wider choices: which modes to use beyond words and a picture, whether to speak words or write them, the learner's translation between representations, and an alternative mode where one is inaccessible. No pattern page instantiates this principle yet; the multimedia-learning pattern instantiates the narrower one.

Default design, while the relationship is untested

This is the earlier page's guidance, kept as a concrete default a designer can act on and revise. Each step is a design proposal, with its evidence stated beside it. No claim here tests the whole sequence, and none tests gesture, demonstration or sign as instructional modes.

  1. Name what each mode must carry. Write one line for each mode: what part of the idea it shows that the others cannot (a diagram shows the spatial arrangement, the narration names the causal steps, the equation gives exact relations). Drop any mode that has no line. Untested as a procedure; from the earlier page ("each mode should add something meaningful"). Relevant graphics added to text have support (see dual coding [+M]); decorative ones are handled on Multimedia Learning.
  2. Choose modes from the content and the task, not from learner type. Spatial content gets a spatial representation for every learner. Do not sort learners by a style inventory or build parallel "matched" versions. Basis: learning-styles matching [-M], against the style-matched version.
  3. Start with words plus one visual type. Add a second visual type only once learners can read the first. A second type of representation helps only under conditions tied to learners' skill at connecting representations (a review recorded under multiple representations [~M]). If learners cannot read the first representation, teach how to read it before adding another.
  4. Speak the words that explain a moving or system-paced visual. Keep as text the words learners must re-read. Narration over an animation is the default. Terms, definitions, formulas and code stay on screen. Basis: modality [~M]. Its benefit is strongest under system pacing and shrinks when learners control the pace.
  5. Do not duplicate full narration as on-screen text over a visualization; key terms are acceptable. Basis: redundancy [~M]. The same review reports the opposite where there is no visualization, and reversals for older learners, learner-paced delivery and abridged text. So captions remain an access layer (step 7), not an error.
  6. Have learners translate between representations at least once per new idea. For example, they sketch the field an expression describes, explain what a simulation shows in words, or point to the part of the diagram a sentence names. Basis: in one undergraduate physics course, sketching and simulation tasks alongside the formula beat calculation-only tasks on a performance test (d = 0.40), with higher reported cognitive load (multiple representations [+M]; one quasi-experimental field study). Self-explanation across representations is under Further evidence and not yet read against this model.
  7. Give every essential meaning a route in a mode each learner can use. Examples are captions or a transcript for audio, a text description for a diagram, and a spoken or signed version of key text. Untested; from the earlier page ("modes should widen access").
  8. Judge the design on a task that needs the relation, in the mode the objective needs, at the horizon the objective sets. Recognising a term on a quiz is not evidence that a learner can read the graph. Untested proposal.

Fitting the design to a situation

The three facts that most change the decision

  1. Representational competence and prior knowledge of this content. A learner who cannot read a representation gains nothing from it and pays its load. Extra representations may be redundant for a learner who already has the model. If the brief does not say, ask: "Can these learners already read a graph (or diagram, or schematic) of this kind and say what it shows, unaided?"
  2. Pacing. Who controls the flow decides whether to speak or write words beside a visual. If the brief does not say, ask: "Will learners watch at a fixed pace (live lecture, broadcast video) or can they pause, replay and re-read?"
  3. Access to each mode. Language proficiency, hearing, vision and the device all limit which modes carry meaning for this learner. If the brief does not say, ask: "Are the learners working in a second language? Does anyone need captions, descriptions or sign? What will they view it on?"
If the brief says… Then change… Basis
Learners are novice on this content and unfamiliar with its representations; or advanced Novice: use words plus one visual type. Teach how to read it first (what the axes, arrows or symbols mean), add one explicit translation prompt per idea, and hold back a second visual type until learners can read the first. Advanced: cut to the most efficient representation, drop duplicating modes, and use translation tasks for checking rather than teaching. Novice: multiple representations [~M] (the review ties the benefit to representational competence). Advanced: untested proposal (the claim pages discuss expertise reversal; no entry here tests it for modes)
Learners are child (primary school) readers Pair text with a relevant graphic, picture or flow diagram. Do not sort children by a style inventory. Dual coding [+M]: the meta-analysis effect held across grade levels (elementary to adult); learning-styles matching [-M]: no style × modality interaction with 5th graders
The goal is a principle or concept in a mathematical or physical domain (how a quantity behaves) Add a dynamic or manipulable representation (simulation) and a learner-made sketch beside the symbolic form. Budget for higher effort while learners work. Multiple representations [+M]: one undergraduate electrodynamics course; untested with children or outside physics
The goal is a procedure or physical action (skill-performance) Show it (a demonstration or video) with spoken steps, and have learners perform it. Keep a short written step list to consult. untested; from the earlier page (demonstration and gesture are untested here)
online-self-paced, learner-paced media On-screen text beside a static visual is acceptable. Give pause, replay and a text version. Do not convert all text to audio. Modality [~M]: its advantage shrinks when learners control pacing
Live lecture, broadcast or other system-paced video with animation Narrate the explanation over the animation. Keep only labels and key terms on screen. Avoid full-sentence duplicate text. Modality [~M]; redundancy [~M]
Learners work in a second language (adult L2 readers) Keep the text available alongside narration and video rather than removing it as redundant. Monitor comprehension of the words alone. Video with narration and text [+M]: one randomized study of Chinese-speaking university EFL learners, immediate test only; redundancy [~M] reports reversals for abridged text
Assessment is open-ended or high-stakes Assess in a format that asks learners to use the representation (explain, predict, sketch), not only true/false recall. Decide in advance whether diagrams may be consulted at the test. Dual coding [+M]: graphics helped more on open-ended and mixed-format tests than on true/false ones
Deaf, hard-of-hearing or blind learners, or no sound or small screens available Give every essential meaning a second accessible route: captions and transcripts, text descriptions of visuals, signed or spoken versions. Test the materials on the device learners will use. untested; from the earlier page ("attention to access"); captions count as accommodation, not redundancy
One session only, or no teacher present Use one representation pair, not a full set. Build the translation prompts and their model answers into the material, so a learner without a teacher can check a sketch or explanation. untested proposal

Seven of the ten rows rest on a claim (the expertise row only for novices). Most of those claims come from one study or were read from abstracts, and each row names the population it was tested with. The advanced half of the expertise row and the procedure, access and one-session rows are untested proposals or the earlier page's guidance.

Observation, state and explanation

Record what the learner did with each representation under stated conditions: which modes they had, who controlled pacing, whether text could be re-read, and what they were asked to produce. Reading a value off a graph, explaining what a diagram shows, translating between representations and using the idea in a new task are different performances. "Visual learner" is a preference or a label, not an observation, and a stated preference does not show which mode helps the learner learn.

Integration across representations, representational competence, channel overload and access barriers are explanatory hypotheses. Scores, reported effort, time on each representation and what a learner points to are observations. None of them directly measures an integrated model.

Response Plausible explanations Observation that could change the interpretation
Says "I'm a visual learner" and asks for video instead of text A stable preference with no learning consequence; a real access need (reading in a second language, a reading difficulty); earlier text materials were poorly designed Give the same content in both forms and compare what the learner can explain afterwards; check whether reading the words alone, in an accessible form, is the barrier.
Can work with each representation alone but cannot connect them (cannot say which part of the graph the equation describes) Low representational competence; the representations were never explicitly mapped; too many representations at once Ask the learner to translate one element (point to it, sketch it); give one mapping prompt and re-probe; remove one representation and see if the link appears.
Follows a narrated animation but cannot recall or use the key terms Spoken words were transient; listening in a second language; the terms need re-reading and were never shown Offer the same segment with key terms on screen or with pause control, and compare.
Reports high effort and does no better with the added modes Duplicated words competing with the visual; an added mode carries nothing new; system pacing too fast Remove one mode at a time (duplicate text first) and compare the explanation produced and the reported effort; vary one feature only.
Does equally well in any mode Prior knowledge of the content; a test too close to the words; the content does not need more than one mode Give a harder changed-condition question; record prior knowledge; if a single mode suffices, use it.

These rows are proposals and have not been tested with learners as diagnostic branches. A single contrast does not identify a cause, and probing can itself change what the learner knows. Record what remains unresolved.

Evidence and qualifications

  • Multiple representations improve learning [+M]: the most direct test here. A quasi-experimental field study (intact recitation groups, n=81, first-year undergraduate electrodynamics) added sketching and interactive simulation tasks alongside the formula and compared them with calculation-only tasks. The intervention group scored higher on a post-intervention vector-field test (p = 0.04, d = 0.40, as printed), with larger normalized gains on conceptual and representational items and higher reported extraneous, intrinsic and germane load. The same page's narrative review (abstract) concludes that a second type of visual representation helps only under conditions tied to students' skill at connecting representations. One course and one immediate post-test; no delayed horizon; not settled against its sources.
  • Dual coding improves learning [+M]: a meta-analysis of 39 experiments (2,103 participants, elementary school to adults) comparing text with graphics against text alone on reading comprehension found g = 0.39, 95% CI [0.26, 0.51], with no difference among pictures, pictorial diagrams and flow diagrams. The effect was larger on open-ended and mixed-format tests than on true/false ones. A further experiment on the page challenges the dual-coding mechanism for picture superiority, not the effect. Partly checked (1 of 2 entries pass, abstract). It covers words with graphics only, not speech, gesture or simulation.
  • Narration rather than on-screen text beside graphics [~M]: a meta-analysis of 43 independent effects (no pooled size in what was read) found narration plus visuals beat on-screen text plus visuals. The benefit was moderated by element interactivity, system- versus learner-pacing and field of study, and was strongest under system pacing. An experiment with 146 college students on a lightning animation agrees. Not settled against its sources. It decides which channel carries the words; it does not show that more modes help.
  • Redundant on-screen text [~M]: a review of 63 studies found that written text duplicating the narration of a visualization most often impaired learning. Adding written text to narration with no visualization had positive effects. Direction reversed for older learners, learner-paced delivery and abridged text. No pooled effect size; not settled against its source. It limits "more modes" to modes that do not compete for one channel.
  • Learning-styles matching does not improve learning [-M]: a systematic review found almost no studies with the crossover design needed to test matching, and judged the one candidate unconvincing. A randomized study with 5th graders (125 classified, 34 in the key analysis) found no style × modality interaction (F(1,38)=1.16, p=0.29). It bears against the style-matched version of multimodal instruction, not against choosing modes by content. Not settled against its sources.
  • Video with narration and on-screen text [+M]: a five-week randomized three-group study with 98 Chinese-speaking university learners of English. On the immediate test, video with narration and text produced higher macrostructure comprehension than text alone or narration with text (r = .33 and .32, as printed). One study, immediate horizon, not yet checked against its source. It qualifies a blanket redundancy rule for second-language readers; it does not test delayed retention.

Retain learners, assignment, materials, pacing, comparator, instrument and horizon when transporting these observations. Do not rank them by effect label: a pooled g, a field-study d and an r from a three-group ANOVA answer different questions. No claim here tests gesture, demonstration, sign or translanguaging as modes. The earlier page's sources on those (Holmström & Schönström 2018; Givens et al. 2020) and its one multimodal-versus-unimodal experiment (Gellevij et al. 2002) have no claim page, so nothing is said here about what they found.

The evidence by question

Each question below links a general claim that gathers the studies behind it into one table, one row per distinct study, with each study's design and effect as recorded and no averaged verdict.

Further evidence, not yet read against this model

Claims this page cited before it was rewritten as a conditional model. They are evidence about the relationship, but none has yet been re-read against the model above; each says how far its own sources have been checked.

Objective and learner-valued goal

Ask what the learner wants to be able to do and why, separately from the designer's objective. Agreement, divergence and uncertainty are observations. A learner who chooses the video has not shown that they value understanding, and a learner who skips the diagram may not be able to read it. Choose a representative task, criterion, permitted aids (may the diagram or transcript be consulted?) and horizon. Naming a term, reading a value off a graph, explaining what a representation shows and acting on it in a real setting need different evidence.

For example, an adult in a first-aid course may want to recognise what to do from what they see at a scene, while the designer's objective is a written quiz on the order of steps. A text quiz cannot show whether the learner can read a scene. A demonstration with spoken steps, followed by an enacted scenario, keeps both aims, and the enacted scenario is the assessment that tests the capability the learner values.

What would revise this model?

The expectation should weaken if comparable learners, on aligned tasks that need the relation, show no advantage of coordinated multiple modes with translation prompts over a single well-designed mode under defensible comparisons. It should also weaken if an immediate advantage is absent at a delayed test. If access (language, hearing, vision, device) explains a learner's difficulty better than the number of modes, change the access route rather than adding modes. If duplicated text helps the learners in front of you, as the second-language and learner-paced findings suggest it can, keep that observation rather than deleting text by rule. If a crossover study with a sound design ever shows a style × modality interaction, the learning-styles qualification should be revisited. Do not protect the model by relabelling each failure as "too many modes" or "low competence" after the fact.

A learner's preference for a mode, immediate comprehension, the ability to translate between representations, delayed retention and use in a real setting are separate claims. The present evidence does not establish an optimal number of modes, a best mode for any learner, or effects of gesture, demonstration or sign.

Claims

Examples

Key Sources

  • Gellevij, M., Van der Meij, H., de Jong, T., & Pieters, J. (2002). Multimodal versus unimodal instruction in a complex learning context. The Journal of Experimental Education, 70(3), 215-239. https://doi.org/10.1080/00220970209599507
  • Holmström, I., & Schönström, K. (2018). Deaf lecturers' translanguaging in a higher education setting: A multimodal multilingual perspective. Applied Linguistics Review, 9(1), 90-111. https://doi.org/10.1515/applirev-2017-0071
  • Givens, M., Holdsworth, L., Mi, X., Rascoe, F., Valk, A., & Viars, K. E. (2019). Multimodal information literacy in higher education: Critical thinking, technology, and technical skill. In Handbook of research on integrating digital technology with literacy pedagogies (pp. 97-120). IGI Global. https://doi.org/10.4018/978-1-7998-0246-4.ch005
  • Lin, L.-F. (2016). The Impact of Video-based Materials on Chinese-Speaking Learners’ English Text Comprehension. English Language Teaching, 9(10). https://doi.org/10.5539/elt.v9n10p1
Page metadata
Field Value
type principle
id multimodal-instruction
aliases integrate-video-narration-text-for-l2-macs-comprehension
title Multimodal Instruction
description For a learner who cannot yet explain a structure that one mode shows poorly, coordinated modes that each carry part of the meaning, with the learner translating between them, may improve immediate comprehension over a single mode, qualified by representational competence, pacing, redundancy and access; matching modes to learning styles does not.
status review
canonical True
generated by claude/unspecified on 2026-10-02
trust unverified — no one has confirmed this page's content
sources 4 — gellevij-2002, holmström-2018, givens-2019, lin-2016

Frontmatter is the page's machine-readable record — see the Schema & Guide.