Design considerations for a future rigorous evaluation of no-deadlines approaches¶

Strategy · All strategies
Evidence · 2 claims (1 for, 1 mixed) · 1 study (1 causal), q2 · 0 of 1 report an effect size · 2 claims rest on one study

Description¶

The memo recommends five areas for a future evaluation: enhancing the synthetic controls (more predictive covariates, cross-directorate multi-unit analyses), reconsidering analytic decisions (annual versus quarterly intervals, thresholds for mid-year switchers, PI anticipation of NDL via Dear Colleague letters), resolving data issues (close-date accuracy, program linkages, partial-NDL programs like CISE), weighing an RCT in which NSF "could randomly select a subset of programs that have not yet implemented NDL," and adding outcomes such as portfolio diversity and staff workload measures.

Design Implications¶

Context¶

Requirements¶

• A feasibility study of deadline frequency across programs before moving to finer time intervals
• Collaboration with NSF to refine comparison-program lists and program linkages

Constraints¶

• An RCT would require waiting several years for impacts and may require randomizing a relatively large number of programs; programs may prefer to choose whether and when to implement NDL
• Including all programs across directorates as comparisons led to poor matches

Target Learners¶

• NSF Evaluation and Assessment Capability staff and future evaluation teams

Target Learning Goals¶

• Rigorous estimation of no-deadlines policy impacts on proposal volume, quality, burden, collaboration, and funding requests

Claims¶

• In exploratory synthetic control analyses, two GEO no-deadlines programs received 120 to 130 fewer proposals relative to their synthetic counterfactuals [+W]
• Synthetic control counterfactuals performed poorly for BIO NDL programs because few comparison programs were available [~W]

Related Strategies¶

Examples¶

-

Key Sources¶

• Fesler, Lily and Lindsay Fox. (2022). No-deadlines Synthetic Control and Exploratory Outcomes Analysis and Recommendations for a Future Rigorous Evaluation [Memorandum]. Alexandria, VA: National Science Foundation. https://www.mathematica.org/publications/no-deadlines-synthetic-control-and-exploratory-outcomes-analysis-and-recommendations-for-a-future