Exploratory analyses suggest NDL increased requested funding, reduced reviewer burden, and shifted proposal quality ratings for two GEO programs¶

Claim · All claims
Evidence · 1 study (2 entries) · causal r1–r2 · q2
Currency · newest study 2022 · not yet checked against its sources · not reviewed by a person

Subclaims¶

q2 i? The two GEO NDL programs requested $265,000 to $294,000 more funding, required 0.9 to 1.8 fewer reviewers, and were less likely to be rated fair (by 4 to 9 percentage points) or underperforming (by one percentage point) relative to counterfactuals. → Fesler 2022 q2 i? There were not consistent patterns for number of collaborative proposals per project or exceptional reviews across the two programs. → Fesler 2022 (2)

Evidence¶

Fesler 2022¶

Fesler, Lily and Lindsay Fox. (2022). No-deadlines Synthetic Control and Exploratory Outcomes Analysis and Recommendations for a Future Rigorous Evaluation [Memorandum]. Alexandria, VA: National Science Foundation. https://www.mathematica.org/publications/no-deadlines-synthetic-control-and-exploratory-outcomes-analysis-and-recommendations-for-a-future

q2 · i? · causal · r1

Same exploratory synthetic control analysis of the two GEO NDL programs; the memo reports the programs "requested $265,000 to $294,000 more funding" and "required 0.9 to 1.8 fewer reviewers" relative to their synthetic counterfactuals, with no significance tests.

"Model estimates suggest that NDL programs received 123 to 128 fewer proposals, requested $265,000 to $294,000 more funding, required 0.9 to 1.8 fewer reviewers, and were less likely to be rated as fair (by 4 to 9 percentage points) or underperforming (by one percentage point)."

Fesler 2022 (2)¶

Fesler, Lily and Lindsay Fox. (2022). No-deadlines Synthetic Control and Exploratory Outcomes Analysis and Recommendations for a Future Rigorous Evaluation [Memorandum]. Alexandria, VA: National Science Foundation. https://www.mathematica.org/publications/no-deadlines-synthetic-control-and-exploratory-outcomes-analysis-and-recommendations-for-a-future

q2 · i? · causal · r2

In the same exploratory analysis of the two GEO NDL programs, the memo reports no consistent patterns for collaborative proposals per project or for exceptional ratings, and conflicting evidence on whether differences increase or decrease after the first year.

"There were not consistent patterns for number of collaborative proposals per project and exceptional reviews."

Discussion¶

Related Claims¶

• In exploratory synthetic control analyses, two GEO no-deadlines programs received 120 to 130 fewer proposals relative to their synthetic counterfactuals — related
• Synthetic control counterfactuals performed poorly for BIO NDL programs because few comparison programs were available — related