Media comparison studies produce uninterpretable "no significant difference" findings.¶

Claim · All claims
Evidence · 3 studies · 2 theoretical r?, 1 review r? · q2–q3
Currency · newest study 2001 · not yet checked against its sources · not reviewed by a person

A "media comparison study" treats an entire delivery medium (e.g., distance vs. face-to-face instruction) as the independent variable and student achievement as the dependent variable. This design confounds learner characteristics, specific media attributes, instructional strategy, and underlying learning theory into a single undifferentiated "treatment," which makes the near-universal "no significant difference" result these studies produce uninterpretable rather than reassuring.

Subclaims¶

q3 i? Media comparison studies are fundamentally confounded because they fail to isolate specific media attributes (type of information representation, sensory modality, level of realism, feedback timing) as separate variables, instead treating the whole delivery medium as one functional unit. → Clark 1983

q2 i? Media attributes — not whole media categories — are the level at which a taxonomy of instructionally relevant properties can be meaningfully defined and tested. → Levie and Dickie 1973

q2 i? A "no significant difference" result is an inconclusive null finding, not evidence of equivalence — much as a "not guilty" verdict does not mean "innocent" — yet researchers and administrators have repeatedly misinterpreted decades of such findings (collected in Russell's widely cited "No Significant Difference" compilation) as proof that delivery medium does not matter for learning. → Lockee, Moore, and Burton 2001

Evidence¶

Clark 1983¶

Clark, R. E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445–459. https://doi.org/10.3102/00346543053004445

q3 · influential peer-reviewed narrative synthesis of decades of media-comparison research · i? · the abstract prints no effect size; the full text may · n=n/a (literature synthesis) · review · r?

Clark's widely-cited synthesis argues that media do not directly cause learning gains; any apparent effect is confounded with the instructional method delivered through the medium, novelty effects from a new technology, and uncontrolled differences in the populations being compared. He concludes that comparing whole delivery media ("Web-based" vs. "face-to-face") tells researchers nothing about why an outcome occurred, because the actual instructional strategy used within each medium is left unmeasured and uncontrolled.

Levie and Dickie 1973¶

Levie, W. H., & Dickie, K. (1973). The analysis and application of media. In R. Travers (Ed.), The Second Handbook of Research on Teaching (pp. 858–882). Rand McNally.

q2 · foundational taxonomic/theoretical chapter · i? · not an effect-size claim; no effect size printed · n=n/a · theoretical · r?

Levie and Dickie define media attributes as "the properties of stimulus materials which are manifest in the physical parameters of media," and provide a taxonomy including type of information representation (text, image, sound), sensory modality addressed, level of realism (abstract to concrete), and capacity for feedback (overt, covert, immediate, delayed). They argue comparison studies should target these specific attributes rather than whole media categories, and concluded as early as 1973 that most learning content can be conveyed through a variety of different media — making whole-medium comparison studies "fruitless."

Lockee, Moore, and Burton 2001¶

Lockee, B. B., Moore, M., & Burton, J. (2001). Old concerns with new distance education research. Educause Quarterly, 24(2), 60–62. Republished in R. West (Ed.), Foundations of Learning and Instructional Design Technology. EdTech Books. https://edtechbooks.org/lidtfoundations/old_concerns_distance_education

q2 · peer-reviewed methodological critique applying Clark (1983) and Levie & Dickie (1973) to distance-education research · i? · no source text available to check; the entry prints no effect size · n=n/a · theoretical · r?

The authors apply Clark's and Levie & Dickie's critiques directly to early-2000s distance-education comparison research, arguing that the same confounds (learner characteristics, media attributes, instructional strategy, absence of any tested learning theory) that discredited earlier media-comparison studies were being repeated under a new name. They highlight the specific fallacy of treating "no significant difference" as proof of equivalence, and recommend that researchers either isolate specific media attributes/strategies as variables, or — when no generalizable theoretical question is actually at stake — conduct local program evaluation rather than claim generalizable research findings.

Studies bearing on this claim¶

Does putting instruction on a digital medium change learning by itself, or only through the method, time and teaching it carries? This table gathers the studies behind this claim and the 14 narrower claims that bear on it: 15 distinct studies (6 quant-synthesis, 4 theoretical, 3 causal, 2 review), q2–q3, 4 of 15 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

This claim is methodological rather than an effect-size finding: it does not assert that any specific instructional medium is more or less effective than another, but that the comparison design itself cannot answer that question validly. The practical implication for instructional designers and researchers is to avoid framing evaluation questions as "does medium A work better than medium B," and instead isolate the specific instructional strategy, media attribute, or learner characteristic actually under investigation — or, when only checking whether a specific local implementation succeeded, to conduct program evaluation rather than present the result as generalizable research.

A live tension worth flagging: the claim that "media do not influence learning, only method does" (the strong form of Clark's position) has itself been contested by media researchers such as Kozma (1994), who argued some media genuinely afford instructional methods that would be impractical or impossible in others — the confound Clark identifies is real, but does not necessarily prove media are inert.

Related Claims¶

• A theoretical gap exists between delivery-system/media theory and instructional strategy theory and other ISD steps — related
• The authors of one physics-course study attribute its technology-supported gains to the student-centred, active-engagement pedagogy the technology was used within — related
• Bilingual education evaluations have produced uninterpretable data by failing to model child-input by treatment interactions — a narrower finding that bears on this claim
• Simulation Based Education Improves Outcomes — related
• No significant interactions emerged between text medium and any other variable (test time, complexity, engagement) — a narrower finding that bears on this claim
• The measured advantage of computer-based instruction over conventional teaching shrinks when the same teacher teaches both versions — related
• A tool's effectiveness results from the whole configuration of events, activities, and contexts in which it is used — a broader claim this one bears on
• Coaching can be effective in all delivery modes: face-to-face, distance, and a combination — related
• General composing heuristics produced mixed results in studies of their effectiveness — a narrower finding that bears on this claim
• Digital drawing shows mixed results relative to paper-based drawing, with a meta-analysis favoring traditional media for some assessment types — a narrower finding that bears on this claim