Comparing learning intervention effects¶

Design Method · All design methods
Evidence · 1 claim (1 for) · 1 study (1 associational), q2 · 0 of 1 report an effect size · 1 claim rests on one study

The wiki's i0–i3 codes are legacy Cohen-like bins for a printed statistic. They are not a scale of educational value or a prediction for an individual learner. In particular, an effect below d = .20 is not necessarily negligible for a broad achievement measure. Preserve the estimate, direction and uncertainty alongside its code.

Comparison record¶

For each effect, identify the following from the study, or write unknown. A pooled mean may combine unlike studies; inspect the underlying effects before assigning shared properties to the mean.

Dimension	Question
Learner state	Who was sampled? What starting capability and relevant characteristics did they have? Were the arms comparable at baseline?
Assignment	How were conditions assigned? What attrition, confounding or contamination could alter the contrast?
Activity and context	What did the intervention and comparison groups actually do, where, and for how long?
Objective and target	What specific capability and observable change were intended?
Outcome instrument	What was measured? How close was the instrument to the taught task? Could it detect the target change? Did ceiling, floor or range restriction matter?
Effect construction	What statistic, direction, standardization denominator and uncertainty interval were reported?
Horizon	When was the outcome measured, and was transfer or persistence assessed?

Two effects are numerically comparable only to the extent that their learner populations, contrasts, outcomes, measures, denominators and horizons are compatible. Unknown is a finding about the available record, not an assumption that the groups or measures match. A local design forecast additionally needs an observed initial state and an explicit goal criterion for the learner in question.

Kraft's bounded reference class¶

Kraft (2020) proposes empirical bins for causal pre-K–12 intervention studies using standardized achievement outcomes: small below .05 SD, medium .05 to below .20, and large .20 or above. These are distributional reference points for that class, not effect forecasts or universal significance thresholds. The IES methods guide documents different distributions for broad standardized tests, narrow tests and specialized researcher-developed tests.

evidence-scales.json records the legacy and Kraft scales separately. python3 scripts/impact_benchmarks.py --summary audits the standardized mean differences recorded in observations/; --json lists what each still lacks. The context for one effect lives on that effect's observation, because population, arms, outcome, timing and result are already recorded there. An optional impact_context block adds only what the record does not carry:

observations:
  - id: reading-posttest
    # ... outcome, time, result (measure_type: hedges_g, estimate, ci_lower, ci_upper) as usual
    impact_context:
      verified_against_source: true
      population_band: pre-k-12        # postsecondary | adult-workplace | mixed | unknown
      outcome_class: standardized-achievement   # researcher-designed | course-assessment | self-report | behavioural | other | unknown
      unit: single-study               # homogeneous-synthesis | heterogeneous-synthesis | unknown
      assignment: credible-quasi-experiment     # only needed when the design family is quasi-experimental
      learner_start: baseline score and arm balance as reported
      measure_alignment: broad reading test
      score_range: no reported ceiling or floor analysis
      standardization_denominator: control-group pretest SD

The controlled values for population_band, outcome_class, unit and assignment are defined in evidence-dimensions.json at the wiki root, with the rest of the shared vocabulary for learners, goals, conditions and outcomes. The script derives assignment for randomized designs, and the metric, value, interval, horizon, comparator and objective, from the record itself, so none is written twice. observation_lib validates the block, and an unknown is a legal value. A claim page is never read for this: its sources[] is rebuilt by sync_evidence_codes.py, and one study backs several claims.

This block illustrates field shape only; none of its sample values describes a particular study. verified_against_source: true requires the original source and the particular effect to be checked. Missing fields, an unreviewed synthesis, a researcher-designed test, a different outcome, or a negative effect receives no automatic Kraft tier. The script does not overwrite the legacy i code. A separate versioned migration can follow review of the eligible records.

Claims¶

In the full samples, NWEA data show far smaller math recovery (.015 SDs) and negative ELA change (-.008 SDs) than ERS (.045 and .019 SDs) [+W]

Page metadata

Field	Value
`type`	`method`
`id`	`impact-evidence-comparison`
`title`	`Comparing learning intervention effects`
`description`	`Check the learner, contrast, intended outcome, instrument and timing before interpreting or comparing a reported effect size.`
`status`	`review`
`trust`	unverified — no one has confirmed this page's content
`sources`	2 — `kraft-2020`, `lipsey-et-al-2012`

Frontmatter is the page's machine-readable record — see the Schema & Guide.