7
Overarching Themes, Key Recommendations, and Chapter Recommendations

OVERARCHING THEMES AND KEY RECOMMENDATIONS

Strategic Direction and Strategic Planning

The panel noted that, despite impressive outcomes, ITL appears to need a more structured strategic plan, with new projects appearing to be primarily driven by legislation and executive orders. Concerns were raised about future staffing levels owing to potential retirements, which could spread available resources too thin and reduce ITL’s ability to deliver broader and more impactful outcomes. Although some strategic vision was evident, it needs clearer and more systematic development and efficient collaboration both within and outside NIST. Developing a well-structured strategic plan would holistically align ITL’s diverse projects with current and future trends, external demands, and emerging topics, helping to consolidate efforts and enable the efficient use of resources.

The panel also found that criteria for defining, prioritizing, and evaluating projects were sometimes not sufficiently clear, and ITL’s overwhelming project demands and budget constraints limit its capacity for in-depth project work. Improved coordination, mentoring, and cross-group collaboration are needed to align projects with ITL’s mission and optimize resource use.

Key Recommendation 1: The Information Technology Laboratory should create a structured strategic plan based on its overarching vision to concentrate its efforts and resources on the most critical areas of work. This will help avoid initiation of projects that are misaligned with the division’s strategic goals and prevent the dilution of resources, ensuring greater impact.

Metrics and Stakeholder Relevance

The panel observed that ITL currently measures its accomplishments based on outputs such as the number of papers, patents, and meetings, rather than outcomes such as impacts on U.S. commerce, the economic scale of supported ecosystems, or the frequency of algorithm usage. Such outputs are easier to quantify but may not impress appropriators. ITL would benefit from focusing on and communicating the tangible impacts on U.S. industry and sharing industry use case stories with legislative staff. Similarly, for ITL’s extensive support of federal agencies, collecting and sharing use case stories would be more effective in communicating impacts than merely reporting the number of publications.

The panel strongly believes that the division’s work appeals to a broader audience beyond its typical stakeholders. While ITL engages with NIST’s Public Affairs Office, it is unclear whether current communication channels adequately highlight ITL’s work to external stakeholders. Researchers engage with the academic community and the Department of Commerce, but a clear strategy for broader external communication could be impactful. Key questions to address include the following: Which additional communities should ITL’s work reach (e.g., Congress, industry, education, or citizens)? What positive

outcomes could arise from broader engagement (e.g., increased funding or better access to resources)? What are the most effective communication channels (e.g., events, videos, textual content)? What is ITL’s web presence strategy, and what resources are needed to optimize communication outcomes?

ITL’s work appeals to a broader audience, but current communication channels may not effectively reach external stakeholders. The panel suggests improving visibility to various communities (e.g., Congress, industry, academia) and enhancing communication strategies, including web presence. Additionally, while dissemination metrics focus on reach, there is a need for metrics that measure and communicate impact to stakeholders and appropriators and that these be included in the strategic plan.

ITL needs to develop metrics that better assess and communicate the impact of its projects to stakeholders and appropriators. Considering constraints on surveying stakeholders, ITL might explore alternative metrics, such as tracking external contributions to ITL documents or reported issues by adopters. Ideas from the open-source community, like those outlined in the Linux Foundation’s “Measuring Your Open Source Program’s Success” could be useful.¹ Additionally, measuring the percentage of repeat collaborating companies could indicate industry value, with different implications for small start-ups versus large technology firms. Years ago, NIST did contract some NIST impact studies (NIST 2023). These studies might be a useful template for ITL to measure impact.

Key Recommendation 2: The Information Technology Laboratory (ITL) should develop impact metrics to be applied uniformly across all of its work. Metrics should, whenever possible, include both the economic benefits for adopters and measurable reductions in risk. These metrics should illustrate the impacts and outcomes of ITL’s work rather than simply providing outputs. Plans for improved communication with ITL’s current and potential stakeholders should be included in the strategic plan.

Artificial Intelligence

The panel believes that artificial intelligence (AI) will significantly impact ITL’s work, with potential opportunities including the use of large language models for scientific and mathematical discovery and enhancing these models. The panel recommends that ITL develop a more ambitious AI strategy focused on critical infrastructures, tools, and methods, and identify key areas for national and international leadership.

Recent advancements in foundational AI and its applications have been revolutionary, and AI is expected to affect nearly all aspects of life and commerce in the coming years. However, its impact on computer security remains uncertain. AI can be used by both attackers and defenders, and the introduction of new AI-driven products and services will bring risks that are not yet fully understood. Additionally, there are growing privacy concerns surrounding the data used to train AI systems. All of this suggests tremendous technological opportunities for ITL.

The panel emphasizes that for ITL to remain effective over the next decade, it must invest in AI staffing, equipment, and expertise. While hiring permanent staff is a long-term solution, establishing a contractor-based or visiting researcher program could be a practical short-term arrangement to enable more agile knowledge transfer. This approach would allow the division to swiftly explore how contemporary AI techniques, such as large language models, can be integrated into existing research workflows.

Cutting-edge research, model training, and AI inference require substantial investment in hardware, data, software, operational resources (such as power), and staff. Building these capabilities will be costly, and attracting top talent will depend on ensuring adequate facilities.

___________________

¹ See the Linux Foundation’s Open Source Guide “Measuring Your Open Source Program’s Success” at https://www.linuxfoundation.org/resources/open-source-guides/measuring-your-open-source-program-success, accessed August 21, 2024.

Key Recommendation 3: The Information Technology Laboratory should enhance its artificial intelligence (AI) expertise to continue being able to have significant impacts in this area. In the long term, this will require adding AI researchers and engineers, either by hiring new talent or by upskilling current staff, or a combination of both. In addition to building a permanent team, the division can create a contractor or visiting researcher program to facilitate flexible knowledge transfer in AI. Such initiatives could also help identify potential candidates for future hiring.

CHAPTER 3: APPLIED CYBERSECURITY DIVISION

Recommendation 3-1: Existing and new Applied Cybersecurity Division projects should include the study of the security, privacy, and responsible uses of artificial intelligence (AI), including the security and privacy characteristics of AI systems.

Recommendation 3-2: The Applied Cybersecurity Division (ACD) should focus on the development of new, specialized cybersecurity guidance for single proprietor or partnership businesses with only a few employees. It should partner with the Small Business Association to develop training materials such as videos and checklists and support regional outreach to enable ACD to have a broader impact within the limited resources available to the program.

Recommendation 3-3: The Applied Cybersecurity Division should conduct a study of the target audiences for its Cybersecurity Framework Profiles to determine if they are being used to full effect, and to determine if their content and format are appropriate for the intended audiences. The Statistical Engineering Division should be consulted on this. Future profiles and the allocation of resources to support their development should be informed by this study.

Recommendation 3-4: The Applied Cybersecurity Division should supplement its work on digital identities with a study of the implications and remediation of a large-scale cyberattack on modern identity systems, such as what might arise from vulnerabilities in widely used desktop or mobile operating systems.

Recommendation 3-5: The Applied Cybersecurity Division (ACD) should explore innovative approaches to staff augmentation and retention. ACD should also develop programs to engage senior volunteer cybersecurity research and engineering talent to serve the nation through its programs and activities.

Recommendation 3-6: The Applied Cybersecurity Division should develop impact metrics for individual projects and apply them uniformly. Metrics should include economic benefits for adopters and quantification of risk reduction, where possible. Useful ideas may be found, for example, through the open-source community and the Linux Foundation.

Recommendation 3-7: The Applied Cybersecurity Division (ACD) should develop and share a strategic vision for how projects are selected and managed in ACD to balance the demands on the division with the available resources and prevent the loss of value and impact from being overstretched.

CHAPTER 4: APPLIED AND COMPUTATIONAL MATHEMATICS DIVISION

Recommendation 4-1: The Applied and Computational Mathematics Division should develop a strategic plan, derived from its strategic vision, to focus its efforts and resources on what have been determined to be the most important lines of work and to prevent the establishment of projects that are not aligned with the strategic vision and that would diffuse the division’s resources and reduce its impact.

Recommendation 4-2: The Applied and Computational Mathematics Division should develop a strategic plan that reflects an integrated vision of the impact of artificial intelligence (AI) on the division, both the short and long term. This plan should address critical questions such as the following:

1. How can AI support and improve productivity for mathematical modeling?
2. Which infrastructures, tools, and methods are critical within this context?
3. What are the key strategic areas of involvement and opportunities for national and international leadership within the AI space?

Recommendation 4-3: The Applied and Computational Mathematics Division should expand the artificial intelligence (AI) expertise available to it. In the long term, it should add AI researchers and engineers. This can be accomplished through new hires, upskilling existing staff, or both. Until it can bring on permanent staff in this area, the division should establish a contractor-based or visiting researcher program to support a more agile knowledge transfer in this domain. These programs might help identify candidates for hiring.

Recommendation 4-4: The Applied and Computational Mathematics Division should designate rooms that its staff can use for remote meetings and remote and in-person conferences with other researchers without the need to schedule them in advance.

Recommendation 4-5: The Applied and Computational Mathematics Division (ACMD) should maintain the Handbook of Mathematical Functions. ACMD should consider whether the division wants to develop new reference materials for the mathematical community.

Recommendation 4-6: The Applied and Computational Mathematics Division (ACMD) should develop a strategy for the improved communication of its work to stakeholders. ACMD should also develop additional metrics to better illustrate the impacts of its ongoing work.

CHAPTER 5: COMPUTER SECURITY DIVISION

Recommendation 5-1: To increase security, automate as much of the workload as possible, and reduce operating costs, the Computer Security Division should migrate access to the National Vulnerability Database to an application programming interface as rapidly as possible.

Recommendation 5-2: The Computer Security Division (CSD) should engage in a strategic planning process to intentionally choose projects that align with its mission and make the best use of the division’s extremely limited resources. This plan should also consider when projects have been successful and what projects ought to be retired to free up resources for new work.

This plan should be clearly communicated to all CSD staff so that they understand exactly how their work fits into the broader divisional mission.

Recommendation 5-3: To free up staff and financial resources for new work, the Computer Security Division should hand off developed technologies to others such as contractors to operate. This will allow researchers to focus on what they are good at and put operations in the hands of those who are skilled at it.

Recommendation 5-4: The Computer Security Division should explore and develop metrics that measure the impact and outcomes resulting from its work rather than simply counting outputs.

REFERENCE