Summary

BACKGROUND AND TASK

Since 1959, the National Institute of Standards and Technology (NIST) has annually solicited the National Academies of Sciences, Engineering, and Medicine to convene expert panels. These panels, comprising professionals from academia, industry, and other scientific and engineering fields, are tasked with evaluating the quality, effectiveness, and resource sufficiency of NIST’s six measurement and standards laboratories. NIST engages the National Academies for these evaluations through an annual contract. For fiscal year 2024, NIST has requested that the National Academies evaluate its Information Technology Laboratory (ITL). As part of this assessment, the panel conducted a site visit, during which they toured the laboratory, held one-on-one discussions with ITL researchers, and followed up with additional inquiries. Leveraging their expertise, the panel reviewed ITL according to the defined scope of work and provided relevant recommendations.

The statement of task includes four key objectives. First, the panel is asked to evaluate ITL’s technical programs, comparing the quality of its research to similar international initiatives and determining whether the programs are sufficient to achieve ITL’s objectives. Second, the panel is asked to assess ITL’s scientific and technical expertise portfolio, considering whether it is world-class and how well it supports ITL’s programs and goals. Third, the panel must review the adequacy of ITL’s facilities, equipment, and human resources in supporting its technical efforts and overarching mission. Last, the panel is asked to evaluate ITL’s effectiveness in disseminating program outcomes, including how well these efforts address stakeholder needs, the comprehensiveness of its dissemination and technology transfer methods, and how effectively ITL monitors stakeholder use and the impact of its outputs.

INFORMATION TECHNOLOGY LABORATORY

ITL comprises six divisions: the Applied and Computational Mathematics Division, the Applied Cybersecurity Division, the Computer Security Division, the Information Access Division, the Software and Systems Division, and the Statistical Engineering Division. These divisions are housed at the NIST campus in Gaithersburg, Maryland; the National Cybersecurity Center of Excellence (NCCoE) in Rockville, Maryland; and the NIST campus in Boulder, Colorado. These divisions contribute to the mission to “promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology through research and development in information technology, mathematics, and statistics” (NIST 2020). This report assesses the Applied and Computational Mathematics Division, the Applied Cybersecurity Division, and the Computer Security Division.

OVERARCHING THEMES AND KEY RECOMMENDATIONS

Strategic Direction and Strategic Planning

The panel noted that, despite impressive outcomes, ITL appears to need a more structured strategic plan, with new projects appearing to be driven primarily by legislation and executive orders. Concerns were raised about future staffing levels owing to potential retirements, which could spread available resources too thin and reduce ITL’s ability to deliver broader and more impactful outcomes. Although some strategic vision was evident, it needs clearer and more systematic development and efficient collaborations both within and outside NIST. Developing a well-structured strategic plan would holistically align ITL’s diverse projects with current and future trends, external demands, and emerging topics, helping to consolidate efforts and enable the efficient use of resources.

The panel also found that criteria for defining, prioritizing, and evaluating projects were sometimes not sufficiently clear, and ITL’s overwhelming project demands and budget constraints limit its capacity for in-depth project work. Improved coordination, mentoring, and cross-group collaboration are needed to align projects with ITL’s mission and optimize resource use.

Key Recommendation 1: The Information Technology Laboratory should create a structured strategic plan based on its overarching vision to concentrate its efforts and resources on the most critical areas of work. This will help avoid initiation of projects that are misaligned with the division’s strategic goals and prevent the dilution of resources, ensuring greater impact.

Metrics and Stakeholder Relevance

The panel observed that ITL currently measures its accomplishments based on outputs such as the number of papers, patents, and meetings rather than on outcomes such as impacts on U.S. commerce, the economic scale of supported ecosystems, or the frequency of algorithm usage. Such outputs are easier to quantify but may not impress appropriators. ITL would benefit from focusing on and communicating the tangible impacts on U.S. industry and sharing industry use case stories with legislative staff. Similarly, for ITL’s extensive support of federal agencies, collecting and sharing use case stories would be more effective in communicating impacts than merely reporting the number of publications.

The panel strongly believes the division’s work appeals to a broader audience beyond its typical stakeholders. While ITL engages with NIST’s Public Affairs Office, it is unclear whether current communication channels adequately highlight ITL’s work to external stakeholders. Researchers engage with the academic community and the Department of Commerce, but a clear strategy for broader external communication could be impactful. Key questions to address include the following: Which stakeholders beyond the usual ones should ITL’s work reach (e.g., Congress, industry, education, or citizens)? What positive outcomes could arise from broader engagement (e.g., increased funding or better access to resources)? What are the most effective communication channels (e.g., events, videos, textual content)? What is ITL’s web presence strategy, and what resources are needed to optimize communication outcomes?

ITL’s current communication channels may not effectively reach external stakeholders. The panel suggests improving visibility to various communities (e.g., Congress, industry, academia) and enhancing communication strategies, including web presence. Additionally, while dissemination metrics focus on reach, there is a need for metrics that measure and communicate impact to stakeholders and appropriators and for these metrics to be included in the strategic plan.

ITL needs to develop metrics that better assess and communicate the impact of its projects to stakeholders and appropriators. Considering constraints on surveying stakeholders, ITL might explore alternative metrics, such as tracking external contributions to ITL documents or reported issues by adopters. Ideas from the open-source community, like those outlined in the Linux Foundation’s

“Measuring Your Open Source Program’s Success,” could be useful.¹ Additionally, measuring the percentage of repeat collaborating companies could indicate industry value, with different implications for small start-ups versus large technology firms. Years ago, NIST did contract some NIST impact studies (NIST 2023). These studies might be a useful template for ITL to measure impact.

Key Recommendation 2: The Information Technology Laboratory (ITL) should develop impact metrics to be applied uniformly across all of its work. Metrics should, whenever possible, include both the economic benefits for adopters and measurable reductions in risk. These metrics should illustrate the impacts and outcomes of ITL’s work rather than simply providing outputs. Plans for improved communication with ITL’s current and potential stakeholders should be included in the strategic plan.

Artificial Intelligence

The panel believes that artificial intelligence (AI) will significantly impact ITL’s work, with potential opportunities that include the use of large language models for scientific and mathematical discovery and enhancing these models. It is advisable that ITL develop a more ambitious AI strategy focused on critical infrastructures, tools, and methods, and identify key areas for national and international leadership.

Recent advancements in foundational AI and its applications have been revolutionary, and AI is expected to affect nearly all aspects of life and commerce in the coming years. However, its impact on computer security remains uncertain. AI can be used by both attackers and defenders, and the introduction of new AI-driven products and services will bring risks that are not yet fully understood. Additionally, there are growing privacy concerns surrounding the data used to train AI systems. All of this suggests tremendous technological opportunities for ITL.

The panel emphasizes that for ITL to remain effective over the next decade, it must invest in AI staffing, equipment, and expertise. While hiring permanent staff is a long-term solution, establishing a contractor-based or visiting researcher program could be a practical short-term arrangement to enable more agile knowledge transfer. This approach would allow the division to swiftly explore how contemporary AI techniques, such as large language models, can be integrated into existing research workflows. Such contractors and visiting researchers can also be a source for new ITL employees over time.

Cutting-edge research, model training, and AI inference require substantial investment in hardware, data, software, operational resources (such as power), and staff. Building these capabilities will be costly, and attracting top talent will depend on ensuring adequate facilities.

Key Recommendation 3: The Information Technology Laboratory should enhance its artificial intelligence (AI) expertise to continue being able to have significant impacts in this area. In the long term, this will require adding AI researchers and engineers, either by hiring new talent or by upskilling current staff, or a combination of both. In addition to building a permanent team, the division can create a contractor or visiting researcher program to facilitate flexible knowledge transfer in AI. Such initiatives could also help identify potential candidates for future hiring.

___________________

¹ The Linux Foundation’s Open Source Guide “Measuring Your Open Source Program’s Success” can be found at https://www.linuxfoundation.org/resources/open-source-guides/measuring-your-open-source-program-success, accessed August 21, 2024.

REFERENCES