Pre-tests reduced adult learners' persistence in a MOOC, though they improved post-test scores among those who completed it¶

Claim · All claims
Evidence · 1 study · causal r? · q3 · n=399
Currency · newest study 2021 · not yet checked against its sources · not reviewed by a person

Pretesting — attempting to answer questions about not-yet-learned material before instruction — reduced persistence in one randomized MOOC experiment. The study measured persistence and post-test learning, not motivation itself; that pre-test failure discourages learners is the proposed explanation (see Discussion), not a recorded finding.

Subclaims¶

q3 i? In a randomized experiment in a real massive open online course, exposure to pre-tests reduced learners' persistence relative to a no-pretest control (completion rates by condition are not recorded here), even though pre-tests improved post-test scores among the learners who did persist. → Janelli & Lipnevich 2021

Evidence¶

Janelli & Lipnevich 2021¶

Janelli, M., & Lipnevich, A. A. (2021). Effects of pre-tests and feedback on performance outcomes and persistence in Massive Open Online Courses. Computers & Education, 161, 104076. doi:10.1016/j.compedu.2020.104076

q3 · randomized experiment (not pre-registered) · i? · no standardized effect size reported · n=399 · causal · r?

399 adults who enrolled in the American Museum of Natural History's climate-change MOOC were randomly assigned to pre-tests without feedback, pre-tests with basic (correct/incorrect) feedback, pre-tests with elaborate feedback, or a no-pretest control. Pre-tests reduced persistence in the course. Across all students, pre-tests and feedback did not affect post-test learning; among those who completed the course, pre-tests improved it. Read as the article's abstract; completion rates by condition are reported in the authors' companion book chapter and are not recorded here.

Studies bearing on this claim¶

What does failing first cost the learner: persistence, belief in the method, the effect of confusion? This table gathers the studies behind this claim and the 2 narrower claims that bear on it: 3 distinct studies (2 causal, 1 review), q2–q3, 0 of 3 with an effect size coded. One row per study, codes as each entry records them; no effect is averaged and no verdict is drawn. Built by scripts/synthesize_claim.py from eval/synthesis/; a row that looks wrong is fixed on the claim it comes from.

Discussion¶

Mechanism. The proposed harm (the recorded study measured persistence, not motivation) runs through learners' affective response to error. When a pretest consists of questions learners cannot answer, repeated failure before any instruction can lower self-efficacy and perceived competence, undermining willingness to engage with the subsequent lesson. This is consistent with motivation theories that treat perceived competence as a driver of engagement — see Autonomy supports intrinsic motivation — and it marks a boundary condition on the otherwise well-supported pretesting effect (see Pretesting enhances learning).

Moderators. The motivational cost plausibly depends on how failure is framed and experienced: whether the pretest is low-stakes, whether learners are told in advance that errors are expected and useful, whether feedback follows quickly, and how difficult the pretest items are relative to learners' background knowledge. Learners with low prior knowledge or low academic confidence are the most plausible at-risk group, since they have the least basis for interpreting pretest failure as informative rather than as evidence of inability. Framing matters in both directions: Conflict-based instruction improves science conceptual learning, though staged contradictions helped only learners who reported being confused, and no study isolates disequilibrium as the mechanism captures the case where early failure provokes productive curiosity, while Belonging interventions improve outcomes suggests that normalizing early error as shared and temporary can buffer the harm.

Tension with the pretesting effect. The literature on pretesting largely documents cognitive benefits — enhanced attention to relevant information and better retention of answers that were initially guessed wrong. The motivational claim does not contradict those findings; it identifies a cost that may co-occur with them. A designer's question is therefore not "pretest or not?" but "how do we capture the error-related learning benefit without the motivational cost?" — for example, by framing pretests as brain-teasers rather than assessments, keeping them short, and avoiding grades or public comparison.

Open questions. The size, durability, and generality of the motivational harm remain unresolved. It is not established whether motivational effects translate into measurable differences in subsequent learning outcomes, or whether they dissipate once instruction begins. The one recorded study (Janelli & Lipnevich 2021) measured persistence, not motivation, and was read from its abstract; studies directly measuring motivation alongside retention in pretesting designs are still needed.

Related Claims¶

• Pretesting enhances learning — the cognitive benefit that this motivational cost trades off against
• Conflict-based instruction improves science conceptual learning, though staged contradictions helped only learners who reported being confused, and no study isolates disequilibrium as the mechanism — productive failure can motivate, but only under the right conditions
• Autonomy supports intrinsic motivation — competence and autonomy are key drivers of the motivation at risk here
• Belonging interventions improve outcomes — framing failure as normal and shared can buffer the motivational harm of early errors
• Instruction can harm learners in ways its performance data do not show. — a broader claim this one bears on
• Retrieval Failure Reduces Benefit — related
• Random measurement noise in pretest scores produces a negative contribution to the correlation between normalized gain and pretest score — related
• Youth motivation to persist in everyday problem-solving is minimal, costing them collaboration and new knowledge — related
• Adult education programs used product-generated learner data minimally, mostly tracking usage time rather than diagnosing struggles — related
• AI assistance reduces persistence: persistence costs concentrate among learners who used AI for direct solutions, not hints — related
• Learners fail to recognize the benefits of pre-instruction testing, ranking it as least effective despite benefiting from it — related
• About half of assigned adults dropped out before post-testing, and non-completers differed initially from completers — related
• Open-ended planning prompts at the start of a MOOC increase course completion by 29% — related
• In chemistry cycle 1, students actively engaged in experimentation still did worse on the volume-rate item, with correct answers falling 6% from pre-test to post-test — related
• Long-term learning gains may be optimized in collaborative situations that display discouraging short-term process costs — related
• MOOCs failed to deliver on their proponents' promises of transforming higher education — related
• Recorded natural-language feedback was associated with lower odds of subsequent persistent failure than no feedback in the randomized course subset — related
• The study addresses whether performance on the low-stakes PISA reading and mathematics assessments can be fostered by coaching and pretesting — related