Introduction

On March 14–15, 2023, the Board on Science, Technology, and Economic Policy (STEP) of the National Academies of Sciences, Engineering, and Medicine held a 2-day workshop in Washington, DC, “Experimentation in Federal Funding.” The purpose of the workshop was to explore the use of data, research, and experiments to improve the processes for and outcomes of federal funding of scientific research.

As delineated in the statement of task, shown in Box I-1, the current system for making funding decisions raises a number of concerns—including, for example, potential biases that may favor more established researchers, less risky proposals, and a narrow applicant pool—and the use of experimentation may help guide any reform of the system. However, because research on the use of experiments in funding research, particularly within government, is limited, the committee charged with planning the workshop (see p. v) decided that it was important to bring together speakers with a wide range of backgrounds and experiences to inform the discussion. The workshop included leading researchers in the burgeoning area of the science of science funding, as well as practitioners from government and the private sector with experience supporting or carrying out experimentation and evaluation. Participants explored the potential of and opportunities for using experimental design to test new federal funding policies and practices, discussed illustrative examples of the use of experimentation from the United States and abroad, considered methods of evaluation, and fostered relationships for future experimentation.

WORKSHOP CONTEXT

Public funding of scientific research has generated enormous scientific, technological, economic, and social benefits, but the core mechanisms by which funding decisions are made can still largely be traced to those implemented when the National Institutes of Health (NIH) was expanded in the late 1940s and the National Science Foundation (NSF) was created in the early 1950s. Numerous concerns have been raised about how the current systems operate, including:

• the time burden of preparing and reviewing research proposals,
• insufficient diversity in the pools of applicants and awardees,
• aspects of the application and selection processes that make it difficult for early-career researchers to obtain adequate funding,
• aspects of the application and selection processes that may make it difficult to obtain funding for potentially breakthrough lines of research, and
• the possibly poor fit of traditional models of peer review with narrowly defined experts in the face of an increasing share of high-impact research emerging from multidisciplinary teams.

Changes in the way that agencies structure grants and solicit and evaluate proposals have the potential to respond to these concerns, and agencies have undertaken some new programs and program modifications intended to address some of these issues. There is, however, an inadequate evidentiary base from which to determine how specific program elements affect specific program goals and outcomes. The burgeoning science of science research has provided many insights, but its capacity to explain how specific program elements affect particular aspects of program outcomes has been constrained by an overwhelming reliance on nonexperimental data. Using experiments, with evaluations of the impact, could provide the evidence needed to systematically improve the achievement of program objectives and address the types of concerns noted above.

BACKGROUND

As detailed in Azoulay and Li (2020), while economists have recognized the key role in science funding on innovation, until recently little empirical work has been done on exploring the relationship between funding mechanisms (and the selection and evaluation criteria used) and scientific discoveries. This is true

despite a body of research of the detrimental impact of current funding selection processes on early-career individuals, reproducibility of results, and novelty of ideas (Alberts et al., 2014). Recent work offers a notable counterexample of experimental evidence on funding outcomes. Teplitskiy et al. (2021) ran an experiment at the Harvard Medical School examining how peer-review scores relate to the novelty of the proposal (as measured by the percentage of key words not previously used). The authors found that the more novel the proposal is, the lower its score. The result is driven by proposals with particularly high levels of novelty. In a related experiment that examines the role of information sharing during peer review, Lane et al. (2021) found that reviewers, when confronted with lower scores than their own, lowered their scores; however, they did not raise their scores when confronted with higher scores. Additionally, while there are few experiments being conducted in the United States, there has been substantial European interest in conducting such experiments (e.g., Heyard et al., 2021; Mengel, 2021; Bendiscioli et al., 2021; Cuello, 2021).

WORKSHOP GOALS AND THIS PROCEEDINGS

Historically, there has been reluctance to “experiment” with public money, but the ongoing concerns about the current system made a National Academies workshop on this topic timely. Indeed, the workshop was held in the context of government-wide emphasis on the use of evidence and data in policymaking, notably the Foundations for Evidence-Based Policymaking Act of 2018 (P.L. 115-435), commonly known as the Evidence Act, which requires, among other things, agency evidence-building and evaluation plans. The workshop speakers, with their wide-ranging backgrounds and experiences, were well qualified to discuss ongoing areas of research on the use of experiments in a variety of contexts, the government context for experimentation and the use of evidence, and concerns about and the potential for the use of experimentation to improve processes for government funding of research.

As STEP chair and planning committee member Adam Jaffe (Brandeis University) said in his opening remarks:

This is an exciting but challenging time to be involved in science and innovation policy and science and innovation research. By planning in advance, you can try things out but do it in a systematic way that ensures—or at least maximizes the chances—that you can learn what works or what doesn’t. . . . Everyone has creative ideas about how to do things differently or how to do things that have never been done before.

Jaffe observed that the science of science funding poses many challenges. Data may be difficult to gather, standard procedures may constrain experiments, or insufficient attention to the establishment of control groups or randomization may limit what can be learned. “You have to figure out how to navigate those issues,” he said.

Workshop presenters and other participants discussed a wide array of topics, as summarized in this proceedings, including examples of experiments by U.S. agencies (Chapter 2), the international context (Chapter 3), nongovernmental perspectives (Chapter 4), challenges with experimentation (Chapter 5), how to structure support for experimentation (Chapter 6), and models for institutionalizing experimentation (Chapter 7), and closing with a roundtable discussion of key ideas from the workshop (Chapter 8). Keynote addresses opened the workshop each day and are summarized together in Chapter 1. The workshop represents an important first step in raising awareness of how experimentation may be used to improve the processes and outcomes of federal funding of research.

This Proceedings of a Workshop was prepared by a workshop rapporteur as a factual summary of what was presented and discussed at the workshop. The planning committee’s role was limited to organizing the workshop. The statements made are those of the rapporteur and do not necessarily represent positions of the workshop participants as a whole; the planning committee; or the National Academies of Sciences, Engineering, and Medicine.

REFERENCES

This page intentionally left blank.