The 1993 Government Performance and Results Act ("the Results Act," or GPRA) requires federal agencies to set strategic goals and to use performance measures for management and budgeting. The objective of GPRA is to encourage greater efficiency, effectiveness, and accountability in federal programs and spending. Development of plans to implement the act has been particularly difficult for agencies responsible for research activities supported by the federal government because of the difficulty of linking results with annual investments in research. The Committee on Science, Engineering, and Public Policy (COSEPUP) of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine is conducting a three-part study to:

1. Identify and analyze the most effective ways to assess the results of research, on the basis of consultation with federal agencies, oversight entities, the research community, industry, states, and agencies of other nations.
2. Help the federal government determine how its agencies can better incorporate research activities in strategic and performance plans and improve the management and effectiveness of research programs, including a determination of what can be reliably measured and the best mechanism for doing so, a determination of what cannot be measured, an evaluation of the extent to which common analytic paradigms could be used across agencies for assessing the results of extramural and intramural research programs and of the training and education of the scientific and engineering

2. workforce, development of such paradigms if feasible, and development of implementation principles and guidelines that could assist Congress and the Office of Management and Budget (OMB) in reviewing agency performance plans.
3. Develop mechanisms to evaluate the effects of implementing GPRA on agency program decisions and on the practices of research, which requires COSEPUP to identify lessons learned and best practices that could be used by other agencies or programs and to determine effective ways for Congress and OMB to use the results of these plans. The committee plans to begin its work on the third charge in the Spring of 1999.

The current study, which addresses parts 1 and 2 described above, began in January 1998. During Part 1 of the study, COSEPUP held three workshops: the first focused on industry methods to evaluate the performance of research, the second on agencies' strategic and performance plans, and the third on methods to evaluate the outcomes of research. Each workshop provided valuable information and a chance for input from all interested parties, including the agencies, the Office of Science and Technology Policy, OMB, congressional committees, the General Accounting Office, disciplinary societies, and researchers. The committee thanks all those who attended for their valuable participation.

The first workshop indicated that applied research is pursued widely in technology-based industries, is focused on achieving well-defined practical outcomes, and is planned and measured in ways similar to ways that other high-risk business undertakings are planned and measured—with business cases, risk-adjusted rate-of-return requirements, and so on. Basic research, where it is still supported by industry, is focused on well-defined areas when such work is judged to be able to lead to technological breakthroughs. Basic research programs also serve as a human resource pool for a firm and as a working contact with similar basic research performed worldwide. Basic research is supported and

evaluated on the basis of how well it serves those purposes, that is, is the work relevant to the firm's future? Is it of high quality? Are the best people recruited to do it?

The second workshop demonstrated the wide range of procedures and methods that agencies have used in responding to GPRA's requirements for planning programs and measuring outcomes. Because applied research programs by definition have desired outcomes directly related to agency missions, evaluating such programs can be relatively straightforward and agencies can use methods similar to those used by industry. We found variations of these methods in place in some agencies and recommend that they be fully and consistently used. It became clear, however, that substantial problems existed for agencies trying to evaluate basic research programs. Urgent concern was expressed that basic research could not be effectively evaluated in the context of GPRA and that misguided attempts to do so could cause great damage. We strongly agree that misguided attempts must be avoided, and we recommend how evaluation can be meaningfully performed.

Since measuring the outcomes from basic research is the most troublesome for the agencies and, in COSEPUP's opinion, the most crucial issue for the long-term economic health of the nation, it has received the most attention in this report. We also deal with measuring the outcomes of applied research and the question of the coordination of planning for all federal research programs.

COSEPUP received encouragement in conducting this study from Representatives F. James Sensenbrenner, Jr., (chair, House Science Committee) and George Brown (member and former chair of the House Science Committee). In a letter (Appendix B) from them, they described the broad range of issues that they see for the planning and evaluation of research in the context of GPRA. This study addresses a portion of those issues, and future work will address other portions.

We have limited this discussion to general principles. Each of our recommendations requires specific guidelines for implemen-

tation, and we have not attempted to provide such guidelines in this report. Because GPRA is in operation today with annual reporting and planning requirements, it seemed important to state general principles as soon as possible and, to the degree that they are acceptable, to direct future work to provide more specific recommendations on implementation. Following the release of this report, COSEPUP will arrange meetings and workshops with agency representatives, congressional staff, and oversight bodies to identify the recommendations they would like COSEPUP to develop further.

This report has benefited from input from various individuals. COSEPUP acknowledges those who made presentations at the workshops and panel discussions: John McTague, Ford Motor Company; Pradip Das, Analytical Sciences Center, Monsanto; Melvin Cohen, Lucent Technologies; David Thomas, Amoco Corporation; Rajen Dalal, Chiron Corporation; Charles Larson, Industrial Research Institute, Inc.; David Rust, George Cooper, Tom Sexton, Bob MacDonald, and Sally Rockey, U.S. Department of Agriculture; Ernest Moniz, Patricia Dehmer, and Robert Astheimer, U.S. Department of Energy; Harold Varmus, Lana Skirboll, John Uzzell, and Wendy Baldwin, National Institutes of Health; Alan Ladwig and Gary Steinberg, National Aeronautics and Space Administration; Neal Lane, Judy Sunley, and Dave Schindel, National Science Foundation; Bob Trew, Joanna Spriggs, and Ed Brown, U.S. Department of Defense; Dani Brzezinska, Fenton Carey, and Paula Ewen, U.S. Department of Transportation; Kevin Teichman, Environmental Protection Agency; Curt Marshall, National Oceanic and Atmospheric Administration; William Ott and Paul Doremus, National Institute of Standards and Technology; T.J. Glauthier and Sarah Horrigan, Office of Management and Budget; Beverly Hartline, Office of Science and Technology Policy; Phillip Kiko, Beth Sokul, and Scott Rayder, House Science Committee; Robert Simon, Senator Jeff Bingaman's office; Genevieve Knezo, Congressional Research Service; Susan Kladiva, General Accounting Office; Henry Small, Institute for Scientific Information; Al Link, University of North Carolina; Irwin Feller, Pennsyl-

vania State University; Paul David, Stanford University and Oxford; Barry Bozeman, Georgia Institute of Technology; and Deborah Boehm-Davis, George Mason University. In addition, COSEPUP would like to thank all those who attended the workshops for their valuable participation

This report has been reviewed by persons chosen for their diverse perspectives and technical expertise in accordance with procedures approved by the National Research Council's Report Review Committee. The purposes of this independent review are to provide candid and critical comments that will assist COSEPUP in making its report as sound as possible and to ensure that the report meets institutional standards of objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following for their participation in the review of this report:

Harvey Bernstein, Barry Bozeman, Rad Byerly, Linda Capuano, Irwin Feller, William Gordon, Paul Gray, Marcelle LaFollette, Stephen Lukasik, Bill Morrill, Ann Peterson, Robert Shannon, Christine Sloane, Bob White, and the report review coordinator, Anita Jones.

Although those persons have provided many constructive comments and suggestions, responsibility for the final content of this report rests solely with COSEPUP.

The production of the report was the result of hard work by the committee as a whole and by the extra effort of the Guidance Group consisting of Peter Diamond, Gerald Dinneen, Mildred Dresselhaus, Ralph Gomory, Philip Majerus, Kenneth Shine, Morris Tanenbaum, and me, cochaired by Ralph Gomory and Morris Tanenbaum, which convened between regular COSEPUP meetings.

The project was aided by the invaluable help of COSEPUP professional staff—Deborah D. Stine, associate director of COSEPUP; Anne-Marie Mazza, senior program officer; Brett Willette, research associate; Carolyn Ryan, research associate;

Michael Hostetler, intern, and its consultants, Susan Cozzens, who provided guidance to the committee on GPRA, and Alan Anderson, science writer. They all helped to draft and revise the report throughout the project. The committee also thanks Richard Bissell, executive director of COSEPUP, who oversaw the committee's activities, and the report's diligent editor, Norman Grossblatt.