The DOE-Office of Science and Technology (OST) has developed an extensive review program with the goal of improving the design, management, and implementation of its programs and technical projects (see Figure C.1). OST seeks to integrate the results of its many reviews to aid managers in decision making. Results may feed into many decisions and activities, such as improving programs and management, formulating budgets, setting priorities, determining technology maturity and availability, and accelerating or decelerating programs. Reviews take place throughout the various levels of the EM organization and can be assigned to three different categories, described below:

• Programmatic reviews are designed to assess the appropriateness and effectiveness of the structure, goals, management, and budget of a particular organization or program. This type of review may be conducted internally (by OST staff) or externally (by standing bodies and groups with EM or OST oversight responsibilities). Reviewers must possess a high level of familiarity with the mission and activities of the organization or program being evaluated.
• Technical assessment reviews are designed to evaluate one or more projects with respect to organizational needs, objectives, responsibilities, or budget, and are used to provide the program or project manager with technical advice and direction. Evaluation criteria include effectiveness in contributing to meeting needs, cost effectiveness, and public and regulatory acceptability. Technical assessment reviews are conducted internally. Reviewers include technology developers and users, who possess program and project knowledge expertise, and knowledgeable stakeholders, for example, regulators.
• Technical peer reviews, also called merit reviews, are used to evaluate a project on its scientific or engineering basis, the competence of the researchers, the soundness of the research plan, and the likelihood of success. Reviewers may consider budgetary aspects of a project only from the standpoint of whether the proposed budget is adequate to complete the work. Technical peer reviews are conducted externally and independently, such that the investigators whose work is being reviewed play no part in the selection or organization of the review panel. From the technical peer review, OST seeks to gain independent, unbiased, technical input or justification for funding project development. This type of review fits most closely the generally accepted definition of "peer review."

These three types of reviews are performed across the organizational levels of the OST, which comprise department, program, and project levels. Reviews conducted at each of these levels are discussed below.

Department-level reviews assess the effectiveness of EM's technology-development program, and all are classified as programmatic reviews. These reviews are submitted to the Assistant Secretary for EM. OST considers department-level reviews to be external in that they are initiated and conducted independently of OST and the focus areas. The following groups conduct department-level reviews:

• U.S. General Accounting Office (Gao). The GAO provides congressionally mandated programmatic oversight to the Department of Energy, and the observations and conclusions from GAO reports are made available to and used by OST.
• National Research Council (NRC)'S Committee On Environmental Management Technologies (CEMT). Until it was discontinued in September 1997, the CEMT² reviewed the broad issues of technology development, implementation, and evaluation within EM. It also evaluated specific technologies that EM considers most important toward reaching its goals. The CEMT submitted two annual reports to the Assistant Secretary for EM (NRC 1995b, 1996). The responsibilities of the CEMT have been assumed by six committees, including the present committee.
• Environmental Management Advisory Board (EMAB). The EMAB provides advice and recommendations to the Assistant Secretary for EM on a broad range of issues relevant to EM. The EMAB is supported by EM and is chartered under the Federal Advisory Committee Act (FACA) and is composed of representatives from tribal, state, and local governments; other federal agencies; environmental and citizen groups; labor organizations; science organizations; and academia. Of the five subcommittees under the EMAB, the Technology Development and Transfer Committee is most directly related to OST.
• Environmental Management Site Specific Advisory Board (EM SSAB). The EM SSAB provides a means for community members to contribute to site-specific policy and technical decisions (e.g., future land use, integrated risk management, resource allocation, and EM priority setting). The board is supported by EM, chartered under FACA and includes all board members from each local EM board throughout the DOE complex. The local boards have unique mission statements, operating procedures, and meeting schedules. Local board members are appointed by the Assistant Secretary for EM and include community members, members from local and tribal governments, and ex officio representatives from DOE, EPA, and state governments.

OST uses program-level reviews to chart progress being made and to assess the suitability of OST and focus area objectives, policies, and plans. These reviews are initiated and conducted within OST and the focus areas, and use both internal and external experts. Program-level reviews include the following:

• OST Board Of Directors Reviews. The OST Board of Directors provides guidance and management for EM's science and technology programs, reviews entire program areas, and weighs high-level policy issues. The board consists of the Deputy Assistant Secretary for OST

• and the field office managers at the Savannah River, Idaho, and Hanford operations offices, and at the Federal (formerly Morgantown) Energy Technology Center (FETC). The OST Board of Directors reviews are classified as programmatic reviews.
• OST Business Reviews. OST business reviews are conducted on a monthly basis and evaluate business management, program design, and execution of technology development activities, as presented by FA/CC program managers. The Deputy Assistant Secretary for OST performs these programmatic reviews.
• Focus Area Mid-Year Reviews. These reviews, conducted for each focus area, assess program management, direction, technical emphasis, and overall soundness of the respective focus area's technology-development program. Both internal and external experts participate as reviewers, which include representatives of private industry, EM user groups, the scientific and academic communities, other federal agencies, and, in some instances, members of the Community Leaders Network (OST's primary stakeholder organization). Mid-year reviews are held in the second or third quarter of the fiscal year, and are classified as programmatic reviews.
• OST Year-End Review. This internal review is an evaluation of individual projects in the context of overall program direction and available resources and is intended to guide EM-50 headquarters staff as they finalize programs for the budget year. In assessing projects, the staff review associated technical peer review reports and consider progress made. The OST year-end review is classified as a technical assessment review.
• Focus Area And Crosscutting Program Reviews. These programmatic reviews assist managers in balancing the respective program's portfolio of technologies to fit the needs of users and stakeholders. Program reviews are conducted by the FA/CC program managers and are attended by technology users, stakeholders, and external industry and academic experts. The results of program reviews feed into EM's strategic planning.

Project-level reviews may be internal or external, and are used to assess proposed or ongoing projects for scientific merit, potential for meeting a site need, potential for risk reduction and safety improvements, cost-effectiveness, regulatory and public acceptability, and commercial viability. Project-level reviews include the following:

• American Society of Mechanical Engineers (ASME) Peer Reviews. The ASME operates under a grant from DOE to conduct technical peer reviews of proposed or ongoing focus area projects. Reviewers are subject matter experts independent of DOE. Review panels are convened to review specific technologies or groups of technologies and are disbanded after issuing their reports. The ASME peer review process is described in further detail in Appendix A.
• Site Technology Coordinating Group (STCG) Reviews. STCG reviews are internal evaluations used to identify and prioritize site technology requirements. STCGs are located at each DOE Operations Office, and are, in general, headed by the Technical Program Officer and composed of site technology users, technology developers, site contractors, and stakeholders. The groups review focus area plans in order to ensure that technology-development decisions address site cleanup needs. STCGs also serve as liaisons to regulators and stakeholders (including local SSABs), and ensure that these perspectives are incorporated into site technology decisions. The work of the STCGs can be categorized as technical assessment reviews.

• Procurement Reviews. Procurement reviews are evaluations of proposals received in response to a solicitation for environmental restoration research and development projects. These solicitations are used when DOE is able to define, but unable to solve, a specific problem. Procurement reviews are conducted by FETC and are considered technical assessment reviews.
• Environmental Management Science Program (EMSP) Reviews. The EMSP is an EM-sponsored research program designed to bridge the gap between fundamental research and needs-driven technology development for environmental restoration. Two types of reviews are conducted as part of the proposal evaluation process for the EMSP. The Office of Energy Research (ER) conducts a peer review to evaluate the merit of proposals received and forwards selected proposals to a panel of EM managers. The EM managers then conduct a technical assessment review to evaluate the proposals' relevance to EM remediation needs.