Synthetic control counterfactuals performed poorly for BIO NDL programs because few comparison programs were available¶
Claim · All claims
Evidence · 1 study · causalr2·q2
Currency · newest study 2022 · checked against its sources 2026-10-20 (automated, 1 of 1 entry) · not reviewed by a person
Subclaims¶
Reading the evidence codes
q is the design tier, i the impact, n the sample, and the kind with its r
the rigour judged within that kind: what each code means.
This claim: no standardized effect is recorded in observations/ for it yet, so no benchmark reading is possible.
q2 i? For the Division of Integrative Organismal Systems Core NDL Program, the estimated counterfactual showed sharp pre-implementation increases and decreases, likely because only four potential comparison programs existed in BIO. → Fesler 2022
Evidence¶
Fesler 2022¶
Fesler, Lily and Lindsay Fox. (2022). No-deadlines Synthetic Control and Exploratory Outcomes Analysis and Recommendations for a Future Rigorous Evaluation [Memorandum]. Alexandria, VA: National Science Foundation. https://www.mathematica.org/publications/no-deadlines-synthetic-control-and-exploratory-outcomes-analysis-and-recommendations-for-a-future
q2 · i? · causal · r2
Assessment of synthetic control fit across the 12 NDL programs in the analytic sample, referencing Figure A10 and Table A10 in the technical appendix. The memo attributes the poor BIO pre-implementation fit to having "very few (only four) potential comparison programs for BIO" and reports BIO programs achieved the worst fit statistics.
"This is likely because there are very few (only four) potential comparison programs for BIO, making it difficult to estimate counterfactuals that closely align their outcomes over the pre-implementation years. Indeed, BIO programs generally achieved the worst fit statistics as well"
Discussion¶
Related Claims¶
- Exploratory analyses suggest NDL increased requested funding, reduced reviewer burden, and shifted proposal quality ratings for two GEO programs — related
- In exploratory synthetic control analyses, two GEO no-deadlines programs received 120 to 130 fewer proposals relative to their synthetic counterfactuals — related
- Nonexperimental methods incorporating pre-treatment outcome measures can replicate experimental ITT impact estimates when control-group crossover occurs — related
Page metadata
| Field | Value |
|---|---|
type |
claim |
id |
synthetic-control-poor-fit-bio-few-comparisons |
title |
Synthetic control counterfactuals performed poorly for BIO NDL programs because few comparison programs were available |
description |
Synthetic control counterfactuals performed poorly for BIO NDL programs because few comparison programs were available |
status |
review |
evidence_strength |
weak |
newest_evidence |
2022 |
source_checked |
2026-10-20 |
generated |
by process:wiki-ingest on 2026-10-08 |
trust |
unverified — no one has confirmed this page's content |
sources |
1 — fesler-2022 |
Frontmatter is the page's machine-readable record — see the Schema & Guide.