Independent Evaluation of EPA’s
Science to Achieve Results (STAR) Research Grants Program
Costel D. Denson, Ph.D.
Professor of Chemical Engineering
University of Delaware
Member, Committee to Review EPA’s Research Grants Program
Board on Environmental Studies and Toxicology
Division on Earth and Life Studies
National Research Council
The National Academies
Subcommittee on Environment, Technology, and Standards
Committee on Science
U.S. House of Representatives
March 11, 2004
Good morning, Mr. Chairman and members of the Committee. My name is Costel Denson. I am a Professor of Chemical Engineering at the University of Delaware where I have been employed since 1977. I have also served there as Vice Provost for Research. In that position I had oversight of, and was responsible for, all aspects of the research enterprise at the university. Recently, I served as a member of the National Research Council (NRC) Committee to Review EPA’s Research Grants Program. The NRC is the operating arm of the National Academies of Sciences and Engineering. I am pleased to be here to discuss the unanimous findings and recommendations of that committee.
The Environmental Protection Agency (EPA) is a mission agency established to protect human health and to safeguard the natural environment. EPA’s regulatory and decision-making role requires that the agency have access to the best available science that is relevant to its mission. In an effort to improve the scientific foundation of its decision-making process, the agency’s Office of Research and Development established the Science to Achieve Results (STAR) research grants program in 1995.
STAR is a competitive, peer-reviewed, extramural research grants program created to encourage interagency collaboration and to increase EPA’s access to the nation’s best scientists and engineers in academic and other nonprofit research institutions. It supports research pertaining to human health and the environment and is designed to maximize the independence of the researchers it supports and to provide an equal opportunity for all researchers to qualify for support.
Over the past six years a number of occasions have arisen where I have had the opportunity to review the STAR Program. When I was chair of EPA’s Office of Research and Development’s Board of Scientific Counselors (BOSC), I oversaw the review that BOSC conducted in 1997 of EPA’s National Center for Environmental Research and Quality Assurance, and the STAR Program, which was operated by that center. Again, in 1999/2000, while still chair of BOSC, I oversaw a review of the STAR program that BOSC and EPA’s Science Advisory Board conducted jointly. And now, as mentioned earlier, I served on the NRC Committee to Review EPA’s Research Grants Program.
The findings and conclusions from this most recent report, along with those from the previous BOSC reports, lead me to conclude the following:
1. Environmental regulatory decisions must be informed by the best science: the STAR program is judged to be the best mechanism that we have for providing the very best science through extramural sources.
2. Research in STAR is focused on EPA’s and the country’s greatest environmental needs.
3. The STAR Program has an exceptional process for the peer-review of proposals. The NRC committee stated in its conclusions that the STAR program “compares favorably with and in some cases exceeds that in place at other agencies that have extramural research programs, such as National Science Foundation and the National Institutes of Environmental Health Sciences (NIH)”.
In 2000, EPA asked the NRC to conduct an independent assessment of the STAR program. In response, the NRC established a committee and gave it the following tasks:
· Assess the program’s scientific merit, its demonstrated or potential impact on the agency’s policies and decisions, and other program benefits that are relevant to EPA’s mission;
· Recommend ways to enhance the program’s scientific merit, impact, and other benefits; and
· In the context of other relevant research conducted or funded by EPA, and in comparison with other basic and applied research grants programs, address the STAR program’s research priorities, research solicitations, peer-review process, ongoing research projects, and results and dissemination of completed research.
In 2003, the committee completed its report, The Measure of STAR, Review of the U.S. Environmental Protection Agency’s Science to Achieve Results (STAR) Research Grants Program.
In its evaluation, the committee developed a series of 9 specific questions that it considered were of greatest importance to the research program. I will present each of these questions, along with the committee’s overall findings and recommendations for the STAR program.
1. Should the STAR program continue to be part of EPA’s research program?
Finding. EPA requires a strong and balanced science and technology research program to fulfill its mission properly. The STAR program is an important part of the overall EPA research program.
The STAR program is EPA’s preeminent program that solicits independent scientific and technologic research from the nation’s best academic and other nonprofit research institutions. The program has established and maintains a high degree of scientific excellence. Through broadly advertised, competitively awarded, peer-reviewed grants, the STAR program provides the agency access to independent information, analyses, and perspectives.
The STAR program provides the agency access to a broad community of researchers, allows it to fund research at the cutting edge of science, and assists it in addressing information gaps that it does not have the internal resources to address properly. The STAR program also encourages its grantees to disseminate their research results widely in peer-reviewed scientific journals.
Recommendation. The STAR program should continue to be an important part of EPA’s research program.
2. What is the unique contribution of the STAR program?
Finding. The STAR program funds important research that is not conducted or funded by other agencies.
For instance, the STAR ecologic-indicators program is the primary source of support of research on the development of water-quality indicators for biologic monitoring. The interdisciplinary centers that STAR has funded also represent an innovative approach to supporting research that will be most relevant for environmental decision making.
The STAR program has also made commendable efforts to leverage funds by establishing research partnerships with other agencies and organizations that have similar or complementary research interests, including the National Science Foundation, the Department of Energy, the Office of Naval Research, the National Aeronautics and Space Administration, the National Oceanic and Atmospheric Administration, the U.S. Department of Agriculture, the National Institute of Environmental Health Sciences, the Department of Interior, and the American Waterworks Research Foundation.
Recommendation. STAR should continue to partner with other government and nongovernment organizations to support research of mutual interest and of relevance to EPA’s mission, explore innovative approaches for carrying out this research, and sponsor a diverse portfolio of research that alerts the agency to emerging issues and provides independent analyses of issues that the agency is currently addressing.
3. Does the STAR program have adequate processes to ensure that it is sponsoring high-quality and relevant research?
Finding. The STAR program has developed a grant-award process that compares favorably with and in some ways exceeds that in place at other agencies that have extramural research programs, such as the National Science Foundation (NSF) and the National Institute of Environmental Health Sciences. An unusually high degree of planning goes into identifying the specific research subjects to be supported. The agency also puts considerable time and thought into preparing effective research solicitations and into funding projects that are relevant to its mission and program needs. Furthermore, the STAR program has established a rigorous peer-review process.
Recommendation. The STAR program should continue to improve the focus of its Request for Applications, and when the agency does not have the capacity internally to adequately define the state of the science in a particular research field, STAR should consider greater use of external experts to assist in identifying the highest-priority research and data gaps.
4. Is the STAR program producing high-quality research results?
Finding. Although it is still too early for comprehensive evaluations of the research results of the STAR program, some STAR research efforts have already substantially improved the scientific foundation for decision-making.
Many STAR projects have resulted in articles in highly respected, peer-reviewed journals—a traditional measure of research quality. These STAR research results have already helped to improve our understanding of the causes, exposures, and effects of environmental pollution—information critical to improving the scientific foundation for decision making. For instance, STAR-funded research on particulate matter has helped to improve our understanding of the biologic mechanisms by which inhaled ambient particles cause health effects and the nature of some of these effects. These data are critical to future regulatory decisions regarding our nation’s ambient air quality.
Recommendation. EPA should continue its efforts to attract “the best and the brightest” researchers to compete for STAR funding.
5. Are the STAR program results useful for EPA decisions and processes?
Finding. The STAR portfolio effectively supports EPA’s mission and research and development strategic plans and GPRA goals. Specific STAR research projects have yielded significant new findings and knowledge critical for regulatory decision making.
The STAR program is too young to be able to document fully the extent to which its research results are being used to inform development of new regulations and environmental-management decisions. However, some STAR projects have already yielded information important for environmental decision making. For example, STAR-sponsored research in endocrine disruptors, particulate matter, and ecologic assessment has resulted in groups of peer-reviewed publications of immediate use in understanding causes, exposures, and effects of environmental pollution. Those results are directly relevant to EPA’s mission to “protect human health and to safeguard the natural environment—air, water, and land—upon which life depends.” For instance, research on ecologic indicators has led to the development of a dynamic, economically linked model to evaluate the driving forces and ecologic consequences of land-use change.
Recommendation. The STAR program and EPA’s Office of Research and Development should develop mechanisms for documenting the extent to which its research is being used to support the agency’s environmental decision making, should consider using outside experts to help document systematically the “state of the science” before research is initiated, and should synthesize the results of the research when it is completed to identify the specific contributions that STAR and other EPA research has made to providing critical information.
6. Is the STAR program effective in providing results relevant to the appropriate audiences?
Finding. The STAR program has been commendably aggressive in experimenting with innovative approaches to communicating the results of its funded research to a wide variety of users and audiences, but its success in these efforts has been uneven.
The STAR program supports research of potential value to a variety of users and audiences, both in and outside EPA. Much of the research is aimed primarily at the scientific community and those responsible for providing technical support for environmental-management decisions.
The program, however, also has other potential users, at least for some of its research results, including other federal agencies; industry; state, tribal, and local governments; nonprofit environmental organizations; and international environmental agencies.
The STAR program has experimented aggressively with a wide variety of communication mechanisms, including EPA’s Web site and publication of reports. Through these efforts, the program appears to substantially exceed the dissemination efforts of most other research-sponsoring organizations, both in and outside the federal government. Nevertheless, the STAR program could substantially improve its dissemination efforts by directing them more effectively to specific users and audiences.
Recommendation. The STAR program should clearly identify the intended audiences for proposed research results as early in the process as possible and indicate them in the Request for Applications. When appropriate, EPA should consider involving representatives of the intended audiences from outside the agency to help define the relevant research results and a strategy for their dissemination.
7. Should the fellowship program continue to be part of EPA’s research program?
Finding. The STAR fellowship program is a valuable mechanism for enabling a continuing supply of graduate students in environmental sciences and engineering to help build a stronger scientific foundation for the nation’s environmental research and management efforts.
It is the only federal fellowship program exclusively designed for students pursuing advanced degrees in environmental sciences.
Recommendation. Given the nation’s continuing need for highly qualified scientists and engineers in environmental research and management, the STAR fellowship program should be continued and funded.
8. Are the STAR program’s funds adequate to achieve its objectives?
Finding. STAR is only able to fund less than 15% of the proposals received for its individual investigator and center grants, and its funding has not kept pace with the rate of inflation.
NIH and NSF strive to fund, on the average, 25-30% of the proposals received. STAR’s budget allows it to fund only 10-15% of the proposals it receives and only about 60% of those rated “excellent” or “very good” by its independent quality peer-review panels. By that measure, STAR does not have sufficient funds to recognize all the best proposals received.
Although the STAR program’s budget grew rapidly in its first 3 years, it has not kept pace with general inflation in the last few years. That is particularly true of the STAR fellowship program. The effect of that budgetary situation is exacerbated by the fact that costs of research have outpaced general inflation for more than a decade. Therefore, at present, STAR funds buy less research than the same amount of money could have bought several years ago.
It is appropriate to consider the funding of the STAR program in the context of the overall funding for all of EPA’s Office of Research and Development, which also has not kept pace with inflation. STAR currently represents about 18% of EPA’s Office of Research and Development total funding. The committee considers that percentage to be a reasonable recognition of the value of independent peer-reviewed research to the agency.
Recommendation. STAR program funding should be maintained at 15-20% of the overall research and development budget, even in budget-constrained times. However, budget planners should clearly recognize the constraints of not having inflation escalators to maintain the level of effort of the entire program.
9. How should the STAR program be evaluated?
Finding. There are no easy answers when it comes to identifying metrics for evaluating research programs, and the best approach for evaluating the STAR program is to establish a structured system of reviews by panels of experts.
The committee assessed the quality, relevance, and performance of the STAR program, as set forth in recent OMB research and development criteria, by using qualitative and quantitative metrics. That is one approach for reviewing the STAR program and similar programs. Several examples of qualitative and quantitative metrics that were used for evaluating the STAR program are: Does the STAR program have a clearly defined plan for regular, external reviews of its research quality, and has this plan been effectively carried out? Has the program made significant contributions to advancing the state of the science in particular research topics? Does the program award grants expeditiously? Does the program have a schedule for the products it intends to produce and how well is it adhering to the schedule?
The committee’s judgment is that quantitative metrics, although outwardly simpler to use, are not necessarily more informative than qualitative metrics. In some cases, quantitative metrics can be misleading, and emphasizing inappropriate metrics can distort the research outputs of a program. Qualitative metrics are less likely to have such effects, but they need to be interpreted carefully.
The committee judges that expert review by a group of people with appropriate expertise is the best method of evaluating broad research programs, such as the STAR program. Expert review is appropriate for evaluating both the processes and the products of the STAR program. Both qualitative and quantitative metrics can provide valuable support for such expert reviews.
Recommendation. STAR and EPA’s Office of Research and Development should establish a structured program of reviews by panels of independent experts and should collect the appropriate information to support these reviews.
Mr. Chairman, that concludes my testimony. I would be pleased to answer any questions. Thank you and the members of your committee for the opportunity to participate in this hearing.