Evidence-Based Teaching and Scientific Reasoning¶

Principle · All principles
Evidence · 38 claims (21 for, 15 mixed, 2 against) · 37 studies (10 review, 9 theoretical, 7 associational, 6 causal, 3 quant-synthesis, 1 qualitative, 1 design), q1–q4 · 0 of 37 report an effect size · 36 claims rest on one study

Conditional relationship¶

This page owns one decision: on what grounds a teacher, designer, school leader or purchaser adopts a teaching approach, programme or product for other people's learners. For a decision-maker (an adult, usually a novice at reading research) whose starting response is to judge an approach by a striking experience in their own class, a colleague's endorsement, a developer's "research-based" label or its fit with a favoured philosophy, the principle says: write the belief as a claim that some result could show wrong, look for a comparison that could have come out otherwise (a randomised or credibly matched comparison, with an outcome aligned to the goal, on learners like these), deliberately look for the case against it, and where no such comparison exists, adopt it as a recorded local trial rather than as settled practice. The expectation is that approaches chosen this way do more for learners, on the outcome that matters to the decision, than approaches chosen by impression, popularity or philosophy.

No claim in the wiki tests that expectation for a teacher's or designer's own decisions. The nearest test is at school level: one federal evaluation of Comprehensive School Reform (Orland 2010; one study behind three claims below) related schools' choice of reform models rated as having a scientific research base to later achievement, with mixed results: low-performing elementary schools gained more in mathematics with better-supported models, elementary reading gains showed no relationship to the model's research base, and only a third of schools chose a rated model at all. The rest of the evidence here supports the principle's premises rather than its outcome: rigorous comparisons do separate programmes that look alike (four first-grade maths curricula); popular practices with a ready story have failed rigorous tests (matching instruction to learning styles; multiple intelligences as a basis for placement); confident impressions of a pattern can persist with nothing behind them (the full-moon belief); and a prompt to consider the opposite is reported to reduce judgment bias. Against it: for adult education and most ed-tech products there is often no rigorous comparison to consult, some programmes cannot be randomised at all, and a null finding has several possible causes.

The relationship is conditional on whether a usable comparison exists for this kind of approach and these learners (early-grade maths curricula: yes; most ed-tech products and adult-education strategies: often not); whether the outcome the evidence measured is the one the decision is about (a state test, a reading posttest, near-transfer, delayed-retention); how the comparison groups were formed (volunteer schools and grades leave selection open); and the horizon (a three-year school gain is not a five-year one).

How this sits beside the converted siblings. Epistemic Cognition holds the general relationship for teaching learners to judge a kind of evidence (teach the standard explicitly, practise it on real claims, justify each judgment; gains stay with the standard taught). When this page's default design is used as professional learning for teachers, that is epistemic cognition with teachers as the learners and study reports as the evidence, and its model applies; the arc below gives the dose for that case without repeating it. Evaluating Sources holds judging a web source's credibility; checking who is behind a developer's study or a vendor's claim is that move. Considering the Opposite is the debiasing step this design uses at the moment of decision. The epistemic half of the art-and-science view in Educational Psychology as Both Art and Science is what this page makes operational.

Default design, while the relationship is untested¶

The earlier pages' guidance (including the folded page "Implement pedagogy based on research and data rather than untested theories that fit a favored philosophy"), kept as a concrete default a decision-maker can act on and revise. Each step is a proposal; its evidence status is stated beside it. Minutes are for one adoption decision by one person or a small team; quantities are untested proposals unless a claim is cited.

State the decision and its outcome (10 minutes). Name the learners (age, stage, starting point), the outcome that would count (an aligned test, a skill observed on a task, attendance, completion), and the horizon (end of unit, end of year, a year later). Untested proposal; without this, no study can be judged relevant.
Write the belief as a falsifiable claim (5 minutes). "Approach X raises outcome Y for learners like ours, compared with what we do now, by the end of the year." Then write what result would show it wrong. If no possible classroom result could, it is not a claim about what works, however confidently held; do not adopt on it. Untested here; from the earlier page (Popper's criterion, via Thornton 2005).
Treat a striking personal experience as a hypothesis (5 minutes). For "it worked brilliantly in my class", write down the cases where it did not work, or the classes where it was not tried, before counting the successes. Confident beliefs about a pattern can persist when the underlying rate is constant (Illusory correlations, like the belief that a full moon affects behavior, persist through confirmation bias despite having no basis in evidence [+M]; a meta-analysis of nearly 40 studies of lunar phase and behaviour, carried here to teachers' beliefs about their own practice, which it did not test).
Look for a comparison that could have come out otherwise (30–60 minutes). Search a clearinghouse such as the What Works Clearinghouse, then systematic reviews, then individual studies; ask a developer for the study behind any "research-based" label and who ran it. Rank what you find: a randomised comparison on similar learners and an aligned outcome first; a matched comparison second, noting how the groups chose to take part; a before-and-after study or testimonials last. Basis: The RCT is considered the gold standard of research evidence because randomization ensures only the intervention could cause treatment-control outcome differences [+W] (a methodological report's stated rationale, not a finding); Educators can more effectively engage with researchers to advance practice by using the What Works Clearinghouse and other sources of research findings as a base [+W] (an article's stated aim; no data). Where randomised comparisons exist, they can separate programmes that all look reasonable: First-grade math achievement is significantly higher under Math Expressions and Saxon Math than under Investigations in Number, Data, and Space and Scott Foresman-Addison Wesley Mathematics [+M].
Make the case against before deciding (10 minutes). Ask one person, ideally from outside the group that already agrees, to argue that the approach will not work here, and list three reasons the opposite conclusion could be right; look specifically for a study that found no effect. Basis: A systematic review of bias-mitigation training names considering the opposite as a previously validated way to reduce judgment biases, without testing it itself [+M] (a systematic review that names the strategy as previously validated on hindsight bias and anchoring, second-hand; untested on teachers' adoption decisions). From the earlier page: discussing an approach only with colleagues who already agree reinforces the bias rather than correcting it.
Do not sort, place or match learners on an untested theory of types. Before adopting an approach that classifies learners (learning styles, multiple intelligences) and changes instruction by type, require research that tests the classification against outcomes, ideally followed over time, before classroom-wide use (the folded page's requirement of "empirical, longitudinal research ... before classroom implementation"). Match the format of instruction to the content (diagrams for spatial content), not to a learner's measured style. Basis: Matching Instruction To Styles No Effect [+M] (a systematic review and a randomised experiment with fifth graders found no style-by-format interaction; its synthesis table counts 4 distinct studies; marked M here because it bears on this page as an example, not as a test of the decision rule); Mi Lacks Empirical Theory Testing [+M] (a review reporting critics, second-hand).
Where no rigorous comparison exists, adopt as a trial, not as practice (one term). Run a small comparison: two sections or classes, one with the new approach and one without; or alternating units; or a staggered start where some classes begin a term later. Write down before starting the outcome, the date, and what result would mean dropping it. For software, the program can assign learners at random and keep the assignment from teachers. Untested proposal as a local procedure. Basis for needing it: There is not yet much rigorous evidence to guide decision making around instructional and support strategies for adult learners [~W]; No readily accessible sources of rigorous evidence exist for most ed-tech products [~W]; Not all intervention programs can be evaluated using an RCT, so non-experimental study designs may be needed to build an evidence base [~W]. That blinded assignment can be run through software: Double-blind experimental design can be successfully implemented in CALL-based pedagogical intervention research [+W] (one vocabulary study's demonstration).
Read a null or a gain before acting on it (20 minutes). A null result may reflect weak implementation, too few learners, an outcome the approach did not target or too short a horizon, not only an approach that does not work: check each before dropping it (A null study finding can have multiple possible causes and requires interpretation before it informs decisions [~W]). A gain in classes that volunteered may reflect the volunteers (The ST Math evaluation's quasi-experimental design cannot rule out selection bias because treatment grades volunteered to participate [~W]).
Record and make it public to someone who can correct it. Keep a one-page record of the decision: the claim, the evidence found, the case against, the local result, and the date to revisit (a year is a reasonable default). Share it beyond those who already agree (a department meeting, another school, a research network). Untested proposal; from the earlier page (Goldacre 2013 on networks; Stanovich 2010 on public, self-correcting knowledge).
Check the five common misreadings by name before accepting or rejecting a piece of evidence (10 minutes; one line each in the record). Each is a move a reader or a learner can be asked to make on any study, report or claim.
- A fair test judged by its wording. "Randomised", "controlled", "research-based" or "evidence-informed" in a description does not make a comparison fair. Ask what was compared with what, how the groups were formed (by chance, by matching, or by volunteering) and whether the outcome was fixed in advance. Basis: The RCT is considered the gold standard of research evidence because randomization ensures only the intervention could cause treatment-control outcome differences [+W] (the logic that makes a test fair is the assignment, not the label); Most surveyed ed tech developers report their products are informed by research, but purchasers struggle to evaluate that research [~W]; The ST Math evaluation's quasi-experimental design cannot rule out selection bias because treatment grades volunteered to participate [~W].
- Public reporting read as evidence. That a practice is published, widely reported, popular, endorsed by peers or used in many schools does not show it works. Public reporting is what lets others check and correct a claim (the earlier page, after Stanovich 2010); it is the start of scrutiny, not its result. Ask whether anyone who reported it tested it, and whether anyone has tried to refute it. Basis: Matching Instruction To Styles No Effect [+M] (a widely held and widely reported practice that failed its tests); More than 70 percent of the 180 most popular literacy applications referenced no research in their design [~M] (popularity without research, as recorded here); Districts rely more on peer recommendations and pilots than on rigorous evidence when choosing ed-tech products [~W].
- Trust read as accuracy. Confidence in a source (a respected colleague, a vendor, an AI chatbot, a fluent textbook) is not a check of what it says. Check a sample of its statements against an independent source before relying on it. Basis: A physics-education review reports one study (Ding et al. 2023) in which high-trust students agreed with ChatGPT's answers 100% of the time although those answers were 82% accurate [+W] (40 undergraduates in physics, reported in a review; carried to other sources).
- A null result read as "does not work" (or a gain read as "works"). Check implementation, number of learners, outcome alignment, horizon and how groups were formed before concluding either way. Basis: A null study finding can have multiple possible causes and requires interpretation before it informs decisions [~W].
- A striking case read as a pattern. Count the cases that did not fit as well as those that did. Basis: Illusory correlations, like the belief that a full moon affects behavior, persist through confirmation bias despite having no basis in evidence [+M].
When the brief names its own assessment, rubric or revision task, run it as given, inside the stated time. If a session or course must end with the brief's own assessment (a certification task, a district rubric, an evidence-review exercise, a task in which contradictory data arrive and learners must revise a conclusion), schedule that assessment in full within the session's time, in the brief's own form and wording, and plan backwards from it. Do not replace it with this page's steps or describe it for later; use this page's moves only as the practice before it, and only for the time left once the assessment is scheduled. Where the task supplies contradictory data, keep that part: learners state their conclusion, receive the new data, revise in writing, and say which earlier claim the data contradicted and why their revision follows (step 2's falsifiable claim, used live). Untested proposal.

Across sessions, when this is taught to teachers. A default arc of five sessions of about 45 minutes across one term (untested proposal; the dose and fading follow Epistemic Cognition, whose evidence is about other learners and other kinds of evidence):

Session 1: each teacher writes one belief from their own practice as a falsifiable claim (steps 2–3); the facilitator models the full-moon example and one of their own.
Session 2: the facilitator walks through one study chosen for the group, thinking aloud: who was compared with whom, how groups were formed, what was measured and when (step 4).
Session 3: pairs read a clearinghouse entry or study the group chose, with a one-page card listing the questions and the five misreadings (step 10); the facilitator only checks answers.
Session 4: each teacher, unaided, plans a one-term local comparison for an approach they want to try (step 7), with the outcome and the dropping rule written before it starts.
Session 5 (end of term): teachers report results, nulls included, to a group from outside the department, and read each other's results with steps 8 and 10.

Where the course has its own assessment, it replaces the session it falls in, not this arc's practice: run it as given (step 11).

Move a teacher from the card to unaided reading once they can name, for two consecutive new studies, the comparison, how groups were formed, and whether the outcome matches their own goal (untested threshold). Teachers who already do this skip to session 4.

When a condition is not met: if there is no time for a search, decide to trial, not to adopt (step 7); if the only evidence is the developer's, say so in the record and set an earlier revisit date.

Fitting the design to a situation¶

The three facts that most change the decision:

Whether a usable comparison exists for this approach and these learners. For early-grade maths curricula there are randomised comparisons; for most ed-tech products and adult-education strategies there are often none, and the decision becomes a local trial. If unstated, ask: has anyone compared this approach with an alternative, on learners like ours, with an outcome we care about?
Who is deciding and how much they can read. A teacher choosing for one class, a district purchasing for many schools and a volunteer tutor need different routes to the evidence (primary studies, clearinghouse summaries, a one-paragraph plain summary). If unstated, ask: who makes this decision, how much time do they have, and have they read a study report before?
What outcome and horizon the decision rests on. A state test at the end of the year, a skill used on the job, completion of a programme and learners' enjoyment are different targets, and evidence for one does not carry to another. If unstated, ask: what result, measured how and when, would make you keep or drop this?

If the brief says…	Then change…	Basis
Decision-makers are `novice` at reading research (most classroom teachers, school leaders)	Start from clearinghouse summaries and a one-page question card (comparison, how groups formed, outcome, horizon); do not ask them to read primary statistics first	untested proposal
Decision-makers are `advanced` (teacher-researchers, a district research office)	Skip summaries; read primary studies; run local trials with random assignment, through software where possible	Double-blind experimental design can be successfully implemented in CALL-based pedagogical intervention research [+W] (one language-learning software study; carried to other settings)
Learners are `child` in early grades and the choice is a maths curriculum	Choose by the randomised comparisons available; weight them above endorsements	First-grade math achievement is significantly higher under Math Expressions and Saxon Math than under Investigations in Number, Data, and Space and Scott Foresman-Addison Wesley Mathematics [+M] (first graders in 39 schools; one cohort)
Learners are `adult` (adult basic education, literacy, ESOL)	Expect few rigorous comparisons; adopt as a one-term local trial with a written dropping rule; carry school evidence only with a note that it was not tested on adults	There is not yet much rigorous evidence to guide decision making around instructional and support strategies for adult learners [~W]
Adults in work settings (training in a company, a health service, a public agency)	Same steps; define the outcome as performance on the job at a stated delay, not end-of-course satisfaction; pilot with one cohort against the existing course	untested proposal; the claims here are about schools and adult education
The choice is an ed-tech product or app	Ask the vendor for a third-party study and who paid for it; a "research-informed" label is not a comparison; without one, trial it against current practice for a term before buying for all	No readily accessible sources of rigorous evidence exist for most ed-tech products [~W]; Most surveyed ed tech developers report their products are informed by research, but purchasers struggle to evaluate that research [~W]; More than 70 percent of the 180 most popular literacy applications referenced no research in their design [~M] (a market scan of children's literacy apps, as recorded here)
The district already decides by peer recommendation and informal pilots	Keep the pilot; give it a comparison class, a pre-set outcome and a dropping rule, so it can come out against the product	Districts rely more on peer recommendations and pilots than on rigorous evidence when choosing ed-tech products [~W] (a survey; describes practice, does not test it)
A whole-school reform model is being chosen for a low-performing elementary school	Prefer a model rated as having a scientific research base; expect a small maths gain over three years, and do not expect the rating to predict reading gains	Low-performing CSR elementary schools that adopted models with scientific research bases gained in mathematics achievement after three years [+W]; No relationship existed between elementary reading achievement gains and selection of a model with a scientific research base [-W] (one evaluation, Orland 2010)
The programme cannot be randomised (whole-school, opt-in, statutory)	Use a matched comparison, write down how schools or classes chose to take part, and report gains as possibly reflecting the volunteers	The ST Math evaluation's quasi-experimental design cannot rule out selection bias because treatment grades volunteered to participate [~W]; Not all intervention programs can be evaluated using an RCT, so non-experimental study designs may be needed to build an evidence base [~W]
High stakes: placement, grouping or tracking by learner type	Do not place, group or match on learning styles or multiple-intelligence profiles; match format to content; require outcome evidence for any classification before using it for placement	Matching Instruction To Styles No Effect [+M]; Mi Lacks Empirical Theory Testing [+M]
A study found no effect	Check implementation, number of learners, outcome alignment and horizon before dropping the approach; record which explanation the study can rule out	A null study finding can have multiple possible causes and requires interpretation before it informs decisions [~W]
Teachers judge how learners feel or engage mainly from what they notice in class	Keep their observation; add one brief anonymous student survey and compare the two before changing practice	Focal math teachers relied primarily on their own observations of student behavior to understand students' feelings about math, supplemented by surveys, with disaggregated data rare [~W] (10 middle-school maths teachers; describes practice)
Learners or teachers use an AI chatbot as a source of answers	Check a sample of its answers against an authoritative source before relying on it, and teach learners to do the same; do not take agreement for accuracy	A physics-education review reports one study (Ding et al. 2023) in which high-trust students agreed with ChatGPT's answers 100% of the time although those answers were 82% accurate [+W] (40 undergraduates in physics, reported in a review)
A single session only (one meeting or one in-service day)	If the brief sets its own end-of-session assessment, schedule it first (step 11); otherwise do steps 1–5 in 45 minutes on one live decision; end with a decision to trial, not to adopt; schedule the review of the trial	untested proposal
A fixed short time budget (20 minutes in a staff meeting)	5 minutes writing the claim and what would show it wrong; 10 minutes reading one prepared clearinghouse summary; 5 minutes for one person's case against	untested proposal
Scarce facilitator time (one coach for many teachers, volunteers, a busy shift)	Teachers work from the question card and a shared template; the coach checks only two things per plan: the pre-set outcome and the dropping rule	untested proposal
The brief names its own assessment, rubric or revision task (a certification task, an evidence-review exercise, new contradictory data that learners must respond to)	Run that assessment as given, in full, inside the stated time; schedule it first and plan backwards; add practice of this page's moves before it only if time remains; keep any contradictory-data revision, with learners revising in writing and naming what the data contradicted	untested proposal
A practice is presented as proven because it is published, widely reported, popular or endorsed	Ask who tested it against what, and whether anyone tried to refute it; treat public reporting as an invitation to check, not as the check	Matching Instruction To Styles No Effect [+M] (a widely reported practice that failed its tests); Districts rely more on peer recommendations and pilots than on rigorous evidence when choosing ed-tech products [~W]
A study is described as "randomised", "controlled" or "research-based"	Check how groups were actually formed and what the comparison was before giving it weight; the wording alone does not make the test fair	The RCT is considered the gold standard of research evidence because randomization ensures only the intervention could cause treatment-control outcome differences [+W]; The ST Math evaluation's quasi-experimental design cannot rule out selection bias because treatment grades volunteered to participate [~W]
The brief names its own routine (a district adoption rubric, an ESSA evidence tier check, a PLC or data-inquiry protocol)	Keep it and its categories; add only what the evidence here bears on: one line stating what result would show the choice wrong, and one comparison or a local trial	untested proposal
Mixed modes: some teachers remote, some in the room, or phone-only	Put the study summary and question card in a shared document read before the meeting; take the case against in writing so remote participants contribute equally	untested proposal
Teachers' own classroom results or student data are private	The teacher decides who sees their trial results; share only class-level summaries with the group; never rank teachers by them	untested proposal
Low confidence or literacy with research (paraprofessionals, volunteer tutors, new teachers)	Replace the card with three spoken questions (compared with what? on whom? measured how?); give one plain-language paragraph per study; no statistics	untested proposal
The result goes to a real audience (a school board, parents, a funder)	Report the comparison, how groups were formed, the outcome and its horizon, nulls included, and what would change the decision; keep the limits in the same document as the result	untested proposal; from the earlier page (public, self-correcting knowledge)
The brief wants a whole programme adopted as designed	Judge and trial the whole programme against the alternative; do not pick components from it and assume the programme's evidence carries to them	untested proposal

Thirteen of the twenty-five rows cite a claim. Only the reform-model row rests on a claim that relates a research-based choice to learner outcomes, and that is one school-level evaluation with mixed results; the curriculum, app, ed-tech and learning-styles rows rest on claims about the evidence for particular options; the others carry a methodological report, a survey or one study to this setting and say so. The remaining rows are untested proposals.

Observation, state and explanation¶

What can be observed is what a decision-maker says and does: the reasons they give for an approach, the sources they name, whether they can state what result would show it wrong, whether they looked for a contrary study, and what their local trial recorded. "Is evidence-minded", "is defending a prior view" and "cannot read research" are inferred states.

Response	Plausible explanations	Observation that could change the interpretation
"It worked brilliantly in my class"	A real effect; a good class or a good term; only the successes were remembered; novelty	Ask for the classes where it was not used or did not work, and the outcome measured; propose a one-term comparison and see whether they accept the dropping rule.
"Everyone in the department agrees"	Shared good experience; a group that has not met a contrary view; deference to a senior colleague	Ask one member to argue the other side in writing; invite someone from outside the department to read the evidence.
Rejects a study whose result conflicts with their experience	The study's learners or outcome really differ; confirmation bias; the study is weak	Ask what result in a study of their own learners would change their mind; if none would, the disagreement is not about evidence.
Accepts a developer's "research-based" label without asking further	Trust in the vendor; no time; does not know what to ask	Give the question card and see whether they can now say what was compared; time spent is the cost to plan for.
Drops an approach after one null study	A real null; weak implementation; too few learners; misaligned outcome	Check the study's implementation and outcome against their own setting before deciding.

These rows are proposals, untested as diagnostics. Asking someone to justify a belief can harden it, so record what was asked and keep the explanations that remain open.

Evidence and qualifications¶

No claim in the wiki tests whether teachers or designers who decide this way produce better learner outcomes than those who decide by impression. The claims below test parts of the relationship or its premises.

School-level test of the relationship, mixed (one study behind three claims). The fifth-year evaluation of the federal Comprehensive School Reform programme (Orland 2010) related schools' choice of reform model to its rated research base. Low-performing CSR elementary schools that adopted models with scientific research bases gained in mathematics achievement after three years [+W]: about 2 percentage points more students proficient in mathematics with a limited research base and 4 with a moderate or moderately strong base; no improvement in non-low-performing schools. No relationship existed between elementary reading achievement gains and selection of a model with a scientific research base [-W]. Only one-third of CSR schools adopted reform models identified as having a scientific research base [-W], so most schools did not choose this way when they could have. All three are coded q2, school-level achievement data from 2002 to 2005, no effect size printed. They describe whether a research base predicted gains, not whether choosing by evidence caused them.
Rigorous comparisons separate programmes. First-grade math achievement is significantly higher under Math Expressions and Saxon Math than under Investigations in Number, Data, and Space and Scott Foresman-Addison Wesley Mathematics [+M]: first graders in 39 schools assigned to curricula (randomized at school level, q3, first cohort); no effect size printed on the claim page. It shows that the choice of curriculum mattered and that only a comparison revealed which; it does not test teachers' decision process.
Popular theories have failed rigorous tests. Matching Instruction To Styles No Effect [+M]: a systematic review found almost no studies with the crossover design needed and judged the one candidate unconvincing; a randomised experiment with fifth graders found no style-by-modality interaction. Its synthesis table counts 4 distinct studies (2 reviews, 2 causal, q2–q3), none with an effect size coded. Mi Lacks Empirical Theory Testing [+M]: a review reports critics' view that few serious theory-testing publications of multiple-intelligences theory exist; second-hand, q2, r1. Both support refusing classification-and-matching approaches without outcome evidence; neither tests what teachers who refuse them achieve instead.
Impressions mislead. Illusory correlations, like the belief that a full moon affects behavior, persist through confirmation bias despite having no basis in evidence [+M]: a meta-analysis of nearly 40 studies found no relationship between lunar phase and behaviour records, while the belief persists. The page's mechanism (confirming cases are noticed and remembered) is the reason for step 3; it is carried to teachers' judgments, not tested on them.
A debiasing move. A systematic review of bias-mitigation training names considering the opposite as a previously validated way to reduce judgment biases, without testing it itself [+M]: a systematic review of bias-mitigation training names "consider the opposite" as a previously validated strategy (hindsight bias, anchoring), but its own synthesis does not test it, and the claim page says it rests on a second-hand report with magnitude unassessed. It supports step 5 as a plausible move, not as a tested part of adoption decisions.
Limit: the evidence often is not there. There is not yet much rigorous evidence to guide decision making around instructional and support strategies for adult learners [~W] (a systematic review, q2); No readily accessible sources of rigorous evidence exist for most ed-tech products [~W] (district survey and interviews); Not all intervention programs can be evaluated using an RCT, so non-experimental study designs may be needed to build an evidence base [~W] (a methodological report's premise). Together they are why the default ends in a local trial rather than "adopt only what an RCT supports": a rule requiring prior rigorous evidence would leave most adult-education and ed-tech decisions with nothing to choose from.
Limit: weaker designs and nulls need reading, not counting. The ST Math evaluation's quasi-experimental design cannot rule out selection bias because treatment grades volunteered to participate [~W] (two evaluations, authors' own stated limitation); A null study finding can have multiple possible causes and requires interpretation before it informs decisions [~W] (a guidance brief, q1).

Learners, settings and outcomes differ across these claims (whole schools, first graders, fifth graders, adults, district purchasers), and almost none prints an effect size, so do not rank them.

The evidence by question¶

When a school or district chooses an ed-tech product, is there rigorous evidence to choose by, and do purchasers use it? No readily accessible sources of rigorous evidence exist for most ed-tech products [~W]

Further evidence, not yet read against this model¶

Claims the earlier page cited, or that bear on the decision, that are used above only for a single step or row, or not at all; none has been read as a test of the model. Each says what it is and how far it bears on this page.

The RCT is considered the gold standard of research evidence because randomization ensures only the intervention could cause treatment-control outcome differences [+W] — a methodological report's stated rationale for randomisation; used above for the ranking in step 4, not a finding.
Educators can more effectively engage with researchers to advance practice by using the What Works Clearinghouse and other sources of research findings as a base [+W] — an article's stated aim (q1, no data); supports starting from a clearinghouse only as a suggestion.
Most surveyed ed tech developers report their products are informed by research, but purchasers struggle to evaluate that research [+W] — a cited survey statistic (90 percent of developers); describes the label, not product effects.
More than 70 percent of the 180 most popular literacy applications referenced no research in their design [+M] — a market scan of children's literacy apps reported in a white paper and recorded here; describes the market, does not test the decision rule.
Double-blind experimental design can be successfully implemented in CALL-based pedagogical intervention research [+W] — a design demonstration from one vocabulary software study; bears on how a local trial can be run, not on whether to run one.
Districts rely more on peer recommendations and pilots than on rigorous evidence when choosing ed-tech products [~W] — a survey of how districts decide (81 percent peer or consultant recommendations against 49 percent rigorous evidence); describes the starting response this page addresses.
Focal math teachers relied primarily on their own observations of student behavior to understand students' feelings about math, supplemented by surveys, with disaggregated data rare [-W] — interviews with 10 middle-school maths teachers nominated as early adopters; describes reliance on personal observation for one kind of judgment; bears on this page only as a description of practice.
A physics-education review reports one study (Ding et al. 2023) in which high-trust students agreed with ChatGPT's answers 100% of the time although those answers were 82% accurate [+W] — a mixed-methods study of 40 undergraduates in physics, reported in a review; bears on trust without checking, for a new kind of source, not on teachers' adoption decisions.
Sequential Field Testing Go No Go [+W]
The RI-CCLPM-C indicates a positive self-concept effect on achievement even when data are generated without any direct effects between self-concept and achievement [+W] — attached 2026-10-19 from Kimmo Sorjonen et al. (2025), which proposed "Do not treat statistically significant cross-lagged effects with good model fit in observational SEM data as strong evidence of causality"; tests this page's relationship.
Time-reversed analysis of simulated data suggests a decreasing rather than increasing effect of self-concept on achievement, yielding inconsistent conclusions from the same data [+W] — attached 2026-10-19 from Kimmo Sorjonen et al. (2025), which proposed "True causal hypotheses should survive testing with a broad set of different analyses; model-choice-sensitive support warrants suspended judgement"; tests this page's relationship.
No active learning intervention study in the Freeman et al. (2014) set or the 2015–2022 sample met all 12 internal validity controls [+W] — attached 2026-10-20 from Amedee Marchand Martella et al. (2023), which proposed "Active learning intervention studies should incorporate internal validity controls such as randomization, group equivalence checks, matched instructors, dosage matching, and implementation fidelity assessment".
Implementation fidelity was assessed in fewer than 8% of active learning intervention studies [+W] — attached 2026-10-20 from Amedee Marchand Martella et al. (2023), which proposed "Active learning intervention studies should incorporate internal validity controls such as randomization, group equivalence checks, matched instructors, dosage matching, and implementation fidelity assessment".
Only four included articles achieved theoretical grounding with empirical validation [+W] — attached 2026-10-21 from Vorderobermeier et al. (2024), which proposed "Interweave theory, didactic instruction, and practice in future EER research and development".
Program-based studies cannot isolate the effect of inquiry-based instruction because they vary many program elements simultaneously [+W] — attached 2026-10-21 from Lin Zhang et al. (2022), which proposed "Educational standards should represent a balanced view of all available research categories rather than relying on one methodology".
Differential attrition may explain the reported reduction in achievement gaps under daily online testing [+W] — attached 2026-10-21 from Robinson (2021), which proposed "Vet widely cited educational claims for internal-validity threats before acting on them, and expose unsupported extravagant claims".
Heterogeneity in teacher program effects was high for stress, psychological distress, and burnout in RCTs, and methodological quality was generally poor across both trial types [+W] — attached 2026-10-21 from Joanne R. Beames et al. (2023), which proposed "Design future teacher mental health program research with methodologically rigorous designs, co-design with teachers, and attention to implementation factors".
Nonexperimental designs, including natural experiments and structural equation modeling, can inform cause-effect relationships to a degree that depends on study design and analysis [+W] — attached 2026-10-21 from Grosz (2023), which proposed "Researchers should discuss practical implications in their articles regardless of design, while transparently communicating limitations and assumptions".
Retrieval practice has been implemented and studied in authentic middle school classrooms, with general recommendations for applied research [+W] — attached 2026-10-22 from Pooja K. Agarwal (2012), which proposed "Apply a century of cognitive psychology research on human learning and memory to real-world education to ground practice in evidence rather than doctrine".
Measurement units for changes in learners are essential to an adequate science of education, since most educational practice problems depend on the amount of some change. [+W] — attached 2026-10-24 from Thorndike (1910), which proposed "Decide competing teaching methods with controlled comparative trials and measurement, rather than from unaided classroom experience.".

Objective and learner-valued goal¶

Here the learner is usually the practitioner making the decision. The designer's (or school's) objective is that adoption decisions rest on comparisons that could have come out otherwise, with the outcome and horizon stated. Elicit separately what the teacher wants: less time wasted on approaches that do not work, the freedom to use professional judgment, something that keeps a difficult class engaged, or not being told by outsiders what to do. Agreement and divergence are both observations. A teacher who values autonomy may hear "use the evidence" as being overruled; a local trial they design themselves serves both their goal and the objective.

For example, a primary teacher may want to keep a phonics game their class loves; the department's objective is a decision based on reading outcomes. Running the game for one term in one class against the existing practice in another, with the reading outcome fixed beforehand and enjoyment recorded as well, tests the objective without discarding what the teacher values. Record both, so that the class's enjoyment is not read as a reading gain, nor a null reading result as a reason to stop something learners value for other reasons.

What would revise this model?¶

The expectation should weaken if schools or teachers who choose approaches by rigorous comparisons, with comparable learners and aligned outcomes, show no better results over several years than those who choose by experience and peer recommendation, as elementary reading did in the reform evaluation. It should change shape if local trials run by teachers turn out too small or too short to tell approaches apart, in which case the default should route decisions to pooled trials across schools rather than to single classrooms. If teachers taught to read research choose differently but their learners do no better, the professional-learning arc serves epistemic goals only. Do not protect the model by calling every failure "the evidence was not rigorous enough" after the fact.

Choosing on evidence, the quality of the evidence chosen on, the outcome for learners and the horizon are separate claims. The present evidence establishes no threshold of evidence that should be required before adoption and no minimum size for a local trial.

Claims¶

Educational Psychology as Both Art and Science
Epistemic Cognition — the general model for teaching learners to judge a kind of evidence; this page's professional-learning arc is that model with teachers as learners
Evaluating Sources — judging who is behind a source, used here on developers' and vendors' claims
Considering the Opposite — the debiasing move used at the moment of an adoption decision

Examples¶

A teacher who notices a technique "worked great" for one class and, before adopting it broadly, checks whether a controlled study backs the impression or looks for a colleague who tried it and got a different result
A department that shares research findings and invites outside speakers, rather than relying solely on internal consensus about what works
Teachers and intervention designers should use mobile devices thoughtfully alongside other teaching approaches, ground designs in existing evidence, and evaluate rigorously
Challenge candidate digital technologies against a learning-based pedagogical framework before adoption
Test the hypotheses of the science of teaching against concrete facts found in school practice

Key Sources¶

Goldacre, B. (2013). Building evidence into education. UK Department for Education.
Thornton, S. (2005). Karl Popper. In E. N. Zalta (Ed.), The Stanford Encyclopedia of Philosophy. Retrieved from https://plato.stanford.edu/entries/popper/
Stanovich, K. E. (2010). How to think straight about psychology (9th ed.). Allyn & Bacon.
Arduini-Van Hoose, N. (2020). The science of educational psychology; The scientific method. In Educational psychology. Retrieved from https://edpsych.pressbooks.sunycreate.cloud. CC BY-NC-SA 4.0.
Peariso, J. F. (2008). Multiple Intelligences or Multiply Misleading: The Critic's View of the Multiple Intelligences Theory. Liberty University. https://eric.ed.gov/?id=ED500515
Huenecke, Dorothy M.; Stansbury, George W. (1975). A Systems Approach to Staff Development. Georgia State Dept. of Education. https://eric.ed.gov/?id=ED123751

Page metadata

Field	Value
`type`	`principle`
`id`	`evidence-based-teaching-and-scientific-reasoning`
`aliases`	`implement-research-based-pedagogy-not-untested-theories`, `field-test-before-full-implementation`
`title`	`Evidence-Based Teaching and Scientific Reasoning`
`description`	For a teacher, designer or purchaser choosing an approach for other people's learners, writing the belief as a claim some result could refute, seeking a comparison that could have come out otherwise and the case against it, and adopting as a recorded local trial where no comparison exists, is expected to select approaches that do more for learners than choosing by impression, popularity or philosophy; no claim in the wiki tests this for practitioners' own decisions, and one school-reform evaluation tests a school-level version with mixed results.
`status`	`review`
`canonical`	`True`
`generated`	by `claude/unspecified` on `2026-10-14`
`trust`	unverified — no one has confirmed this page's content
`sources`	2 — `peariso-2008`, `huenecke-1975`

Frontmatter is the page's machine-readable record — see the Schema & Guide.