How Claims Are Checked¶
Generated by
scripts/build_check_report.pyfrom the repository's own records. Do not edit by hand.
This wiki is written by language models, from published research, and checked by automated steps. No page has yet been reviewed by a person: every page reports its trust tier, and every one says unverified. This page sets out each check a claim passes on its way in and after, what each one catches and cannot catch, and how much of the wiki it has reached so far. What the evidence adds up to is on State of the Evidence.
Health at a glance¶
| Claims | 27,612, from 7,164 sources |
| Reviewed by a person | 0 |
| Every evidence entry compared with its source by the faithfulness judge | 4,886 claims (18%) |
| Claims citing at least one DOI | 20,188 claims (73%) |
| Sources whose citations went through the Crossref identity gate | 6,731 of 7,164 |
| Sources read for impossible statistics and invented references | 6,251 of 7,135 (2026-10-25) |
| Cited DOIs checked for retraction | 6,793 of 6,793 (2026-10-25) |
| Retracted sources found and acted on | 3 |
| Claims past their recheck year (fast-ageing kinds only) | 2,571 of 5,710 with one |
| Claims removed by a check, all time | 4,424 |
The checks, in the order a claim meets them¶
1. Inclusion¶
Each source is screened against the published inclusion criteria before anything is extracted: opinion pieces, out-of-scope work, sources with nothing extractable and sources already covered are refused. Every decision, either way, is logged in sources/manifest.ndjson, so anyone can check whether a given article was reviewed.
Cannot catch. A source that is in scope and badly done. Inclusion is about what a source is, not how good it is.
Coverage. 7,383 sources reviewed: 7,164 ingested, 219 refused (out-of-scope 111, validation-error 83, uncoded 22, no-contributions-extracted 2, parse-error 1).
2. Extraction and structural validation¶
A model reads the article and writes claims with their evidence. A validator refuses any extraction whose structure is broken, whose citation does not match the article, or whose codes are out of range; a refused article contributes nothing.
Cannot catch. A well-formed claim that misreads its article (step 5 looks for that).
Coverage. 84 articles refused by the validator.
3. Citation identity¶
Every DOI written on a page is resolved against Crossref (DataCite after a Crossref 404) and compared on title, first author and year. A DOI that resolves to a different paper is removed, because a wrong DOI reads as verified; one that cannot be settled is flagged for a person. Books, which no registry indexes, are checked by a person against the publisher and recorded in sources/authorities.ndjson.
Cannot catch. A citation with no DOI and no registry record; a correct citation to a study that is itself wrong.
Coverage. 6,731 of 7,164 ingested sources passed the gate with the registry reachable; 224 DOIs removed as the wrong paper, 75 flagged for a person; 3 book citations verified by a person.
4. A verifiable source for every claim¶
A claim whose evidence has no DOI after a Crossref and DataCite search is removed, unless it rests on a report from a named evaluation organisation (What Works Clearinghouse, MDRC, RAND and others) or a principle, pattern or ten or more designs depend on it, in which case it says so in an Evidence quality line.
Cannot catch. Whether the DOI'd source is any good.
Coverage. 20,188 of 27,612 claims (73%) cite a DOI; 2,878 claims removed for having none.
5. Faithfulness to the source¶
A second model, from a different provider, reads each evidence entry beside the article it cites (full text where an open copy exists, otherwise the abstract) and checks every number, direction, population and design the page states. It runs on every claim that three or more design pages cite, and on every new batch. A failure stops the batch until a person or agent has read it; a judge error is dismissed with a written reason.
Cannot catch. A claim checked only against an abstract, for detail an abstract does not carry. The judge is a model and has been wrong; its failures are read before they are acted on.
Coverage. 4,886 claims (18%) have had every evidence entry compared with its source and passed; 158 judge failures were read and recorded with a reason. Each claim states this on its page, in its Currency line.
6. Can the source be believed?¶
Added 2026-10-25, after an audit of AI-in-education meta-analyses found that a study this wiki had made ten claims from reports an effect size of 1.374 (it cannot exceed 1) and cites five ChatGPT studies dated before ChatGPT existed. Each source's text is now read for statistics that cannot be right (a proportion of variance above 1, an ANOVA row whose mean square is not its sum of squares over its degrees of freedom, p values their own test statistics contradict), for references dated before the tool they name, and, for AI-era sources, for reference DOIs that do not exist or point to a different work. Every cited DOI is checked for a retraction, withdrawal or expression of concern. A high-severity finding stops the batch; a confirmed one removes the source's claims.
Cannot catch. Invented references that carry no DOI and no anachronism; errors that leave the arithmetic consistent; a source whose text was not available (abstract only).
Coverage. 6,251 of 7,135 sources read (2026-10-25): 43 with a finding, 4 high severity. Reference DOIs: 6,519 DOIs from 610 AI-era sources resolved; 202 sources have one that does not resolve or names another work, listed for a person but not acted on automatically, because text extracted from a PDF breaks DOIs across lines and most of these are layout, not invention. 7 findings decided: 6 confirmed and acted on, 1 false alarms. 3 retracted sources found; of 6,793 DOIs the wiki cites, 0 were registered at neither Crossref nor DataCite when last checked (2026-10-25).
7. Does the study measure what the claim says?¶
Also added 2026-10-25, from the same audit's most common findings. For each causal claim a model reads the study and reports: whether the outcome is measured performance or only self-report (a claim that learning improved cannot rest on students' ratings of their own learning); what the effect was compared with (a randomised group, an intact class, the same students before, nothing); whether the intervention came bundled with something else that could explain it; and whether students were counted as independent where whole classes or teams were assigned. A finding is acted on by retitling the claim to what was measured, stating the design's limit, correcting a design code that overstates it, or removing the claim.
Cannot catch. Problems the paper does not report. The judge reads only what the study says about itself.
Coverage. causal entries on AI claims: 1,319 of 1,319 judged (2026-10-25); found unit 108, isolation 96, comparison 71. 261 findings read and acted on.
8. Evidence codes¶
Each evidence entry carries a design tier (q1–q4), an impact code (i0–i3, which must rest on an effect size the source prints, otherwise i?), a sample size with its unit, and a kind with a rigour rating judged against that kind's own standard (an interview study and a trial are each rated for what they are). Rigour is coded from the article, never from the wiki's paraphrase of it.
Cannot catch. Whether a study's conclusion is true. The codes describe how a study was done, not whether it was right.
Coverage. 33,179 evidence entries; 4,569 rest on a printed effect size and 28,629 report none (i?); 28,501 carry a rigour rating.
9. Duplicates and merges¶
Two claims stating the same finding are merged, and two pages on the same thing are folded together, only when two models judge them the same; the decisions are kept, so a pair judged different is not raised again.
Cannot catch. Near-duplicates phrased far apart.
Coverage. 2,965 pair decisions recorded.
10. Currency¶
Each claim states the year of its newest study and when its sources were last checked. Claims whose findings age quickly (the accuracy of a tool or model, local or general context such as baselines) carry a recheck year; past it the page says it is overdue rather than hiding it. Design findings carry none: research does not expire on a calendar.
Cannot catch. A finding superseded by newer research the wiki has not ingested.
Coverage. 5,710 claims carry a recheck year; 2,571 are past it.
11. Structure¶
A lint run on every change fails on broken links, misfiled pages, missing identity fields, stale generated files, embargoed research, and any claim citing a DOI recorded as retracted. Nothing merges while it fails.
Cannot catch. Anything about content.
Coverage. Every change to the wiki.
Removed by a check¶
Removals are logged with their reason in eval/removed/pages.ndjson; git history keeps every removed page.
| Reason | Claims and pages |
|---|---|
| No verifiable source (no DOI after a Crossref and DataCite search) | 2,878 |
| About how research is done, not a design decision | 995 |
| Local context only (one district, programme or year) | 217 |
| Source cannot be believed (impossible statistics, invented references, retraction) | 11 |
| Citation chase: a retracted article or off-topic methods literature | 83 |
| Misfiled or thin page folded or dropped | 236 |
| Other (see eval/removed/pages.ndjson) | 4 |
What no check here does¶
- No person has reviewed a page. The trust tier on every page says so.
stablestatus requires a person's recorded review, and no page has one yet. - No check corrects for publication bias. The literature over-reports positive results, and education technology more than most. A claim resting on published studies inherits that.
- Most claims rest on one study. One study in one setting has not been shown to hold elsewhere; State of the Evidence gives the share.
- The checks are models and patterns. They catch what they are built to catch, and each has been wrong; every failure they raise is read before it is acted on, and that reading is recorded.
Every record behind these numbers is in the repository: sources/manifest.ndjson, eval/load-bearing/, eval/source-validity/, eval/removed/pages.ndjson.