_______________

_______________

The American people, through their elected representatives, support the nation's research enterprise in the expectation of substantial returns on their investment: a higher standard of living, a healthier society, an environmentally sustainable economy, and a strong national security. Knowing the power of research in addressing national objectives, the nation has committed itself to a broad set of investments to uphold its research capability.

The National Research Council has already prepared studies that describe the effectiveness of research investments in addressing national concerns. Research investments affect the quality of research done. The present study asks how to evaluate research-leadership status. COSEPUP proposed contributing a set of experiments in international benchmarking. International benchmarking compares the quality and impact of research in one country (or region) with world standards of excellence. The use of international benchmarking was also advocated in 1995 by the "Press report," Allocating Federal Funds for Science and Technology (see appendix B), for the purpose of providing objective information for the executive branch and Congress. The need for objective evaluations has intensified since the passage in 1993 of the Government Performance and Results Act (GPRA), which requires annual performance reports by all federal agencies, including those which support research. GPRA is discussed in the 1999 COSEPUP report Evaluating Federal Research Programs: Research and the Government Performance and Results Act .

Although the use of international benchmarking was not new, it had not been attempted on a scale large enough to contribute to national

policy. Accordingly, in 1997, COSEPUP decided to undertake a set of experiments to test the efficacy of international benchmarking. The committee chose three areas of research—mathematics, immunology, and materials science and engineering—that are quite different from one another in size, funding, numbers of subdisciplines, and other qualities. COSEPUP appointed a panel for each field and sought to experiment with providing information for relatively modest commitments of time and money. Once the panels had completed their reports, the committee held a workshop with agency representatives, congressional staff, and oversight bodies to discuss the findings. (See appendix C for a summary of the workshop.)

This report describes the background, methodology, experimental results, and findings of the international benchmarking experiments and concludes that international benchmarking by a panel of experts can be efficient and reasonably objective. International benchmarking might also be a valuable assessment tool for those seeking to implement GPRA.

Maxine Singer

Chair

Committee on Science, Engineering, and Public Policy

This report is the product of many individuals. First, COSEPUP acknowledges those who made presentations at the Workshop on International Benchmarking: Arden Bement, chair, Panel on Materials Science and Engineering; Irving Weissman, chair, Panel on Immunology; Peter Lax, chair, Panel on Mathematics; and the discussants: Richard Russell, House Committee on Science; Don Lewis, director, Division of Mathematical Sciences, National Science Foundation; Patricia Dehmer, associate director, Office of Science, Office of Basic Energy Sciences, Department of Energy; Helen Quill, chief, Basic Immunology Branch, Division of Allergy, Immunology and Transplantation, National Institute of Allergy and Infectious Diseases, National Institutes of Health; and Irwin Feller, director, Institute for Policy Research and Evaluation, and professor of economics, Pennsylvania State University.

The report was reviewed by persons chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the National Research Council's Report Review Committee. The purposes of this independent review are to provide candid and critical comments that will assist COSEPUP in making its report as sound as possible and to ensure that the report meets institutional standards of objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following for their participation in the review of this report: Arden L. Bement, Jr., Robert M. White, Margaret H. Wright, Paul A. Fleury, Susan Cozzens, Arthur Bienenstock, and the report review coordinator, Anita

Jones, and the report review monitor, John Ahearne. Although those persons provided many constructive comments and suggestions, responsibility for the content of the report rests solely with COSEPUP.

The production of this report was the result of hard work by the committee as a whole and by the extra effort of the Guidance Group, consisting of current and former COSEPUP members Marye Anne Fox (Chair), David Challoner, Ellis Cowling, Gerald Dinneen, Mildred S. Dresselhaus, Alexander Flax, and Ralph E. Gomory.

The project was aided by the invaluable help of COSEPUP professional staff: Deborah D. Stine, associate director of COSEPUP and study director; Rebecca Burka, administrative associate; Aubrey Sabala, project assistant; Alan Anderson, consultant writer; and Norman Grossblatt, editor. The committee also thanks Richard Bissell, executive director of the Policy Division and director of COSEPUP, who oversaw the committee's activities.