Hug and Aeschbach taxonomy of grant review criteria and elements coded as entity-dimension combinations¶

Theory · All theories
Evidence · no claims cited

Description¶

The report describes a comprehensive taxonomy of the criteria and elements reviewers use to evaluate proposals, developed by Hug and Aeschbach (2020) after screening more than 16,000 articles and identifying 12 studies on grant review criteria and elements. The authors "content coded grant elements according to combinations of an 'entity' (an element of the proposal) to be evaluated and the dimension used to evaluate it," identifying 373 entity-dimension combinations across the 12 studies. Most studies (n = 8) included biomedicine, but several included domains relevant to NSF: two natural science, two engineering, and four social science. This entity-plus-dimension structure is the report's own working definition of an element.

Design Implications¶

Context¶

Requirements¶

• Requires coding each grant element as a combination of an entity to be evaluated and a dimension along which it is evaluated

Constraints¶

• Most of the 12 underlying studies included biomedicine, limiting domain coverage

Target Learners¶

• Research funders, program officers, and researchers studying grant review processes

Target Learning Objectives¶

• Classifying and comparing the criteria and elements used in grant proposal review

Claims¶

Related Theories¶

• Five frameworks for evaluating ex ante broader impacts of research vary in emphasis on methods versus outcomes

Examples¶

• Allocate human and AI effort by dimension type: automate explicit-criteria dimensions, retain human review for pedagogical judgment

Key Sources¶

• Chandler, Jesse, Emily Rosen, Kimberley Raue, Katlyn Lee Milless and Danielle Rockman. 2024. A Review of Funder Instructions and Grant Reviewer Practices for Assessing the Intellectual Merit and Other Impacts of Research. Alexandria, VA: U.S. National Science Foundation. https://www.nsf.gov/pubs/2024/nsf24020/nsf24020.pdf