Skip to content

Always report and interpret effect sizes alongside p values for primary outcomes

Principle · All principles
Evidence · 25 claims (17 for, 8 mixed) · 26 studies (10 quant-synthesis, 6 causal, 5 review, 5 theoretical), q1–q4 · 6 of 26 report an effect size · 24 claims rest on one study

Conditional relationship

This page owns one decision: what a report of an instructional intervention must say about the size of its effect, and how a reader should read that size before acting on it. The person here is whoever writes or reads an evaluation: a researcher or graduate student writing up a study, a designer reporting a pilot of their own course, a teacher, leader or purchaser reading a study before adopting a design (an adult, often a novice at reading statistics). Their usual starting response is to treat "statistically significant" as "it works" and "not significant" as "it does not work". The principle says: for every primary outcome, report a standardized effect size with its interval beside the p value, say what was compared with what, on which measure, at which horizon, and interpret the size against effects found for the same kind of outcome, measure and learners, not against a universal label. The expectation is that decisions made this way (adopt, revise, drop, test again) follow the size and reliability of the effect rather than the accident of whether a test crossed p < .05.

No claim in the wiki tests that expectation: nothing here compares the decisions or the later learning outcomes of readers given effect sizes with those of readers given p values alone. What the claims establish are the premises, each in education data:

  • Significance and size come apart. In What Works Clearinghouse re-analyses, author-reported significant effects were no longer significant once clustering and multiple comparisons were corrected for, while one of those effects (0.35 on a reading test) was still large enough to count as substantively important by the Clearinghouse's convention.
  • Filtering on significance inflates the size. A methodological commentary reports that applying the significance filter in education produces large magnitude (Type M) errors; a meta-analysis of test-enhanced learning found that publication-bias corrections cut its average transfer prediction substantially.
  • The size depends on the measure. Across reviews of tutoring systems, early literacy, critical thinking and mindset, effects were larger, or more often significant, on researcher-made or aligned tests than on standardized ones.
  • Universal labels mislead. Effects differ by subject tested and by grade, and shrink as a program scales up, so "0.4 is medium" says little without a reference class.

The relationship is conditional on the reader's decision (adopt a program, revise a design, plan a larger study); the outcome measure (aligned with what was taught, or standardized); the design (randomized or not, individuals or clusters, how many); and whether the reported effect survived a significance filter or a selective search. It rests on the view, set out in Statistical significance as the criterion distinguishing a program effect from chance, that a program "has an effect" when a group difference is significant (A program is said to have an effect when the difference between program and non-program groups is statistically significant [~W]); this page keeps that criterion for chance and adds the size beside it.

How this sits beside the converted siblings. Evidence-Based Teaching and Scientific Reasoning holds the wider decision of adopting an approach on the grounds of a comparison that could have come out otherwise; reading the size of that comparison's effect is one move inside it, and this page gives that move's detail without repeating its model. Validity, Reliability and Bias in Classroom Assessment holds whether a classroom measure measures what it claims; this page takes the measure as given and asks how big the difference on it is. How a study should be designed so its effect can be trusted at all is in Designing a Valid Experiment.

Default design, while the relationship is untested

The earlier page's requirements, made into a routine a writer or reader can follow. Each step is a proposal; its evidence status is stated beside it.

  1. Report a standardized effect size for every primary outcome, with an interval. Give the group means, standard deviations and n, the effect size (Cohen's d or Hedges' g for two groups; name the index), and its 95% confidence interval, every time a p value is reported. From the earlier page and its source (the APA Task Force); the requirement itself is untested here as a cause of better decisions. Where a study reports only test statistics, as one randomized trial of self-regulated learning training reported group-by-time interactions with no effect size per outcome, the reader cannot tell how much each outcome moved (In a randomized trial of SRL interventions in an online mathematics preparation course, group and time interacted significantly on SRL knowledge, self-efficacy and the SRL overall score, with Group TDP (training + diary + peer feedback) showing the most prominent gains [~W]).
  2. Say what was compared with what. Name the comparison condition (business as usual, an active alternative, no treatment, a pre-test), the unit assigned (learners, classes, schools) and the horizon of the outcome (immediate-performance, delayed-retention). An effect size without its comparison cannot be read. Untested proposal.
  3. Name the measure and its alignment. Say whether the outcome is a researcher-made test of what was taught or a standardized test, and report them separately when both exist. Evidence: effects were much larger on local than on standardized tests across tutoring-system evaluations (ITS evaluation results depend strongly on test type: effects are large on locally developed tests (ES = 0.73) and small on standardized tests (ES = 0.13) [+M]).
  4. Correct the significance test for the design, and keep the effect size whatever the test says. In a class- or school-randomized study, test at the level assigned or correct for clustering; report the effect size whether or not the corrected test is significant. Evidence: Greenwood et al. (1993) reported a statistically significant CWPT effect on CTBS-Reading, but the WWC-corrected analysis was not statistically significant [+M].
  5. Read a non-significant result against the smallest effect the study could detect. Report the minimum detectable effect size; if it is larger than the effect that would matter, call the result inconclusive, not "no effect". Evidence: a 50-person, 10-school study reported it could detect only d = 0.81 (The evaluation used a school-level cluster-randomized quasi-experiment (N = 50) that was powered only to detect large effects (d = 0.81) [+M]).
  6. Interpret the size against a reference class, not a universal label. Compare it with effects from studies of the same kind of outcome, measure type, subject and grade, and with earlier studies of the same program. Where a convention is needed for a reader, the Clearinghouse's +0.25 "substantively important" line (about 10 percentile points) is a convention, not a finding (The guide treats an effect size of +0.25 or higher as substantively important, equivalent to raising performance at least 10 percentile points [+W]). Cohen's labels (0.2, 0.5, 0.8) are untested as a guide for education decisions here.
  7. Translate the size into the reader's terms. Add one plain sentence: the improvement in percentile points, or the raw-score difference on the test the reader knows. Never write a d of 0.65 as "65%"; a meta-analysis of self-regulated learning interventions did (Self-regulated learning interventions have a moderate effect (0.65) on learning outcomes in online and blended environments [~W]: the claim page quotes the authors saying the interventions "affect learning results ... by 65%"). Untested proposal, from the earlier page's "practical significance".
  8. Expect a single significant result to be too large. Before acting on one significant study, especially a small one or one found by a search for positive results, discount its size and look for a replication or a pre-registered test. Evidence: Large Type M (magnitude) errors almost certainly exist when applying the statistical significance filter in education research [+M]. The amount to discount is not given by any claim here.

Dose and progression across sessions (for teaching this to graduate students or teachers, the earlier page's target learners). An untested proposal: four sessions of about 60 minutes. Session 1: from one real report, the learner extracts means, SDs and n and computes d by hand (20 minutes), then reads two studies with the same p value and very different d (20 minutes), then writes the one-sentence translation (20 minutes); the teacher supplies the studies and a worked calculation. Session 2: the learner reads a study with a significant author-reported result that a re-analysis corrected (step 4), and a non-significant one with a large minimum detectable effect (step 5); the teacher supplies the pairs, learners write the reading. Session 3: the learner places a reported effect against a reference class (same measure type, subject and grade) the teacher has gathered, then finds one themselves. Session 4: the learner writes the effect-size paragraph for a study of their own choice, or their own pilot, unaided, and a peer checks it against steps 1–8. A learner moves on when their unaided paragraph for a new study covers the comparison, the measure, the interval and the translation; one who still writes "significant, so it works" repeats session 2 with a new pair.

When the routine cannot be followed in full: if the source prints no SDs, report the test statistic, the n per group and that no effect size could be computed; do not estimate one by assumption.

Fitting the design to a situation

The three facts that most change the decision:

  • Who reads the report, and what will they decide with it. A journal referee, a funder, a school board choosing a program and a designer revising a course need the same numbers but different translations and different reference classes. If unstated, ask: who acts on this result, and what are their options?
  • What the outcome measure is. A test written for the intervention and a standardized test give different sizes for the same program, so a size means nothing until the measure is named. If unstated, ask: who wrote the test, and how close is it to what was taught?
  • How the study was designed and found. Clustering, a small sample, adaptive assignment, or a result picked because it was significant each change how far the reported size can be trusted. If unstated, ask: what was randomized, how many units were there, and how was this study chosen from others?
If the brief says… Then change… Basis
The reader is a teacher, leader or purchaser, novice at statistics, or has low confidence with numbers Lead with one sentence in percentile points or raw-score difference on a test they know, then the d and its interval; keep the p value in a note The guide treats an effect size of +0.25 or higher as substantively important, equivalent to raising performance at least 10 percentile points [+W] (a reporting convention, not tested with readers)
The learners are graduate students in educational research (a course on reading studies) Run the four-session arc above; use real published pairs where significance and size disagree as the compared cases untested proposal; the earlier page's source proposes the Task Force report as a teaching framework
The outcome is a researcher-made or aligned test Report it apart from any standardized outcome; compare its size only with other aligned-test effects; expect a smaller effect on a standardized test ITS evaluation results depend strongly on test type: effects are large on locally developed tests (ES = 0.73) and small on standardized tests (ES = 0.13) [+M]; Language and early writing interventions showed larger effects on researcher-developed than standardized outcome measures, while phonological awareness, print knowledge, and decoding effects were comparable across measure types [~M] (tutoring systems and early literacy; the gap is not the same for every skill)
The study randomized classes or schools Test significance at the level assigned or correct for clustering; report the effect size either way; state the number of clusters Greenwood et al. (1993) reported a statistically significant CWPT effect on CTBS-Reading, but the WWC-corrected analysis was not statistically significant [+M]; Clustering effects vary by design but are typically large in education random-assignment evaluations [+M]
A small pilot of one course or class, or a single session only Report the effect size, interval and minimum detectable effect; call a non-significant result inconclusive; plan the next test rather than a verdict The evaluation used a school-level cluster-randomized quasi-experiment (N = 50) that was powered only to detect large effects (d = 0.81) [+M] (pre-service teachers; carried to any small pilot)
The decision rests on one significant study, or on studies found by searching for positive results Discount the reported size; look for a replication, a pre-registered trial or a bias-corrected meta-analytic estimate before adopting Large Type M (magnitude) errors almost certainly exist when applying the statistical significance filter in education research [+M]; Publication-bias assessments substantially reduce the intercept prediction, often indicating no positive transfer when the identified moderators are absent [+M] (test-enhanced learning transfer)
An online platform assigned learners adaptively (a multi-armed bandit or similar) Report that assignment was adaptive and treat the effect size as likely overstated, especially a small one; confirm with a fixed-allocation comparison MAB assignment overestimates effect sizes for normally distributed rewards, with greater overestimation for smaller true effects [+M] (simulation)
A program moves from pilot to wide use (a district, a state) Expect the scaled effect to be smaller than the pilot's; set the adoption threshold against the scaled estimate, not the pilot's Effect sizes of Pre-K Mathematics decreased over time as the intervention moved through successive scale-up phases [~M] (one pre-K mathematics program)
Comparing effects across subjects or grades (a mathematics program against a reading program; primary against secondary) Compare sizes only within the same subject and grade band; do not rank a mathematics effect above a reading effect on size alone Effect sizes based on mathematics tests are significantly larger than effect sizes based on reading tests in targeted secondary interventions [~M]; Interventions in higher Grades tend to have somewhat lower effect sizes; other study characteristics show few significant moderator effects [~M] (struggling learners, targeted interventions)
The report uses a non-standard index (a gain ratio, a percentage gain, a pre-post change only) Report the raw means and SDs as well, name the index and its own threshold, and do not compare it with d benchmarks Within the explicit morphology instruction group, the pre-to-post gain ratio was above Blake's 1.2 minimum for morphological awareness (1.28) and below it for reading comprehension (1.06) [~M] (one EFL study reporting Blake's modified gain ratio from experimental-group pre-post means)
The report says an effect of 0.65 means a 65% improvement, or a reader made that error Use the error as the worked contrast: show what a d of 0.65 means in percentile points and why it is not a percentage Self-regulated learning interventions have a moderate effect (0.65) on learning outcomes in online and blended environments [~W] (the misreading is quoted on the claim page)
Several outcomes reported only as a multivariate test or interaction Give an effect size per primary outcome, with its interval, beside the omnibus test In a randomized trial of SRL interventions in an online mathematics preparation course, group and time interacted significantly on SRL knowledge, self-efficacy and the SRL overall score, with Group TDP (training + diary + peer feedback) showing the most prominent gains [~W] (reported no per-outcome effect size)
The brief names its own reporting template, standard or rubric (a journal's, a funder's, the Clearinghouse's) Keep it and its categories as given; add only what it lacks among: effect size with interval, comparison, measure type, horizon untested proposal
A fixed short time budget (a 45-minute professional-development slot for teachers) One study, 10 minutes extracting the numbers, 15 computing d and the percentile translation, 15 reading it against a supplied reference class, 5 writing the one-sentence verdict untested proposal
Mixed modes: some readers in the room, some remote or on phones; or a self-paced online module Put the numbers in a shared table everyone can see, give the d calculation as a fill-in template, and collect each person's one-sentence translation in writing untested proposal
One facilitator for many teachers, or scarce facilitator time Give a worked example and an answer key so peers check each other's calculations; the facilitator reads only the final translations untested proposal
The audience is a board, funder or the public, who should see the conclusion and not the analysis Give the translated effect, the comparison and the uncertainty in two sentences; put the effect sizes, intervals and test statistics in an appendix untested proposal
Data are from a small group of identifiable learners (one class, a small program) Report effect sizes for groups only; suppress cells too small to protect privacy and say so; the learners or institution decide who sees individual results untested proposal
The brief wants a whole results section or evaluation report written, not separate exercises Keep it whole: draft the full section once, then revise once against steps 1–8 untested proposal

Eleven of the nineteen rows rest on a claim. None tests the reporting move itself on readers' decisions; each shows a reason the move matters (sizes diverge from significance, depend on the measure, shrink with scale) in the settings its study covered, and the rows carry that to the reader's setting. The rest are untested proposals.

Observation, state and explanation

What can be observed is what a report contains (which numbers, which comparison, which measure) and what a reader writes or decides after reading it. "Understands practical significance" and "over-trusts p values" are inferred states.

Response Plausible explanations Observation that could change the interpretation
A report gives p values and no effect sizes The writer was not asked for them; the analysis (a multivariate test, an interaction) has no obvious single effect size; the effect is small and the writer prefers not to show it Ask for the means, SDs and n per outcome and compute d; compare with the significance verdict.
A large effect from a small or single study A real large effect; inflation from the significance filter; an aligned measure written for the intervention Check the measure, the sample, the minimum detectable effect, and whether a larger or independent study found a similar size.
A significant result with a tiny effect A very large sample; a real but practically small effect Translate into percentile points or raw score; ask whether that difference would change the reader's decision.
A non-significant result with a moderate effect size Too few units to detect it; clustering not accounted for; no effect, with a noisy estimate Report the interval and the minimum detectable effect; look for other studies.
A reader concludes "it works" or "it doesn't" from p alone, or reads d as a percentage Habit from training that emphasized significance; no translation given Ask the reader to state the size in percentile points and the comparison condition; see whether their decision changes.

These rows are proposals, untested with readers as diagnostics. A second look at a report can change the reader's reading of it, so record what was asked and in what order.

Evidence and qualifications

No claim in the wiki tests whether reporting and interpreting effect sizes improves the decisions or outcomes that follow. The claims below support its premises in education data, and qualify it.

None of these claims measures what a reader does differently with an effect size in hand. Do not rank them by their own effect sizes; they are about different questions.

The evidence by question

Further evidence, not yet read against this model

Objective and learner-valued goal

The objective this page serves is a decision about a design or program that follows the size and reliability of its effect on an outcome that matters, at the horizon that matters. Elicit separately what the person writing or reading the report values: a researcher may value a publishable significant result; a graduate student, passing the methods course; a purchaser, a clear yes or no; a teacher, whether their own learners will do better. Agreement and divergence are both observations. A writer who values publication may leave out a small effect size; a reader who wants a yes or no may skip the interval. Make the translated size and its uncertainty the shortest sentence in the report, so it is the one most readers see.

For example, a school leader wants to know whether a reading program will help the struggling readers in their own school. The study reports a significant effect on a researcher-made test and a small, non-significant effect on the state test. The objective this page sets is to state both sizes with their measures; the leader's goal is the state test, so the second number is the one that bears on their decision. Record both.

What would revise this model?

The expectation should weaken if studies of readers (teachers, leaders, researchers) given effect sizes with intervals and translations make no better, or worse, adoption decisions than readers given p values alone, judged against later outcomes; or if readers routinely misread effect sizes in ways that cost more than misreading p values. If measure-type gaps disappear in studies with well-aligned standardized tests, step 3 becomes a reporting detail rather than an interpretive rule. If bias-corrected or pre-registered estimates in education turn out close to the significant published ones, step 8's discount should shrink. Do not protect the model by blaming every misread effect on a missing translation.

Reporting an effect size, interpreting it against a reference class, and the decision that follows are separate claims. The present evidence establishes no threshold that makes an education effect worth acting on, no discount factor for a single study, and no reference class that serves every outcome.

Claims

Examples

-

Key Sources

  • Thompson, B. (2000). The APA Task Force on Statistical Inference (TFSI) Report as a Framework for Teaching and Evaluating Students' Understandings of Study Validity. Paper presented at the annual meeting of the American Educational Research Association. https://eric.ed.gov/?id=ED440121
  • Mohamed Farrag Ahmed Badawi. (2019). The Effect of Explicit English Morphology Instruction on EFL Secondary School Students’ Morphological Awareness and Reading Comprehension. English Language Teaching, 12(4). https://doi.org/10.5539/elt.v12n4p166
Page metadata
Field Value
type principle
id always-report-interpret-effect-sizes
aliases report-effect-sizes-with-p-values
title Always report and interpret effect sizes alongside p values for primary outcomes
description For anyone writing or reading an evaluation of an instructional intervention, reporting a standardized effect size with its interval, comparison and measure beside every p value, and reading it against effects for the same kind of outcome rather than a universal label, is expected to lead to better adopt-or-revise decisions than significance alone; no claim in the wiki tests that expectation, though claims show significance and size diverging, significance filtering inflating sizes, and sizes depending on the measure.
status review
canonical True
generated by claude/unspecified on 2026-10-15
trust unverified — no one has confirmed this page's content
sources 2 — thompson-2000, mohamed-farrag-ahmed-badawi-2019

Frontmatter is the page's machine-readable record — see the Schema & Guide.