An Assessment of Selected Divisions of the National Institute of Standards and Technology Information Technology Laboratory: Fiscal Year 2024 (2025)
Chapter: 6 Information Technology Laboratory's Responses to the Recommendations of Previous Assessment Reports
6
Information Technology Laboratory’s Responses to the Recommendations of Previous Assessment Reports
This chapter provides the Information Technology Laboratory’s (ITL’s) responses to recommendations made in the previous assessments of the divisions in this report by the National Academies of Sciences, Engineering, and Medicine in fiscal year (FY) 2018 (NASEM 2018), and some recommendations from the FY 2021 assessment (NASEM 2021). The sections below include observations on specific recommendations from 2018 that were addressed to the offices and divisions.
2018 REPORT
Staffing and Recruitment
Recommendation: The [Applied Computational and Mathematics Division] (ACMD) should evaluate its organizational and recruiting practices in order to better meet the challenges it faces. Ideas that should be considered include the use of contractors to broaden the pool of potential participants in the ACMD mission; the use of sabbatical opportunities for career staff to broaden the range of skills in response to new areas for ACMD; and development of a more effective pipeline for graduate students into ACMD through, for example, a broad-based university affiliates program.
ITL response: We have greatly increased our pool of participants. During the period October 2022–December 2023, we
- Hosted 9 NIST/NRC postdoctoral associates
- Engaged with 12 postdoctoral or senior researchers through the PREP program, contracts, or grants
- Supported 2 technicians via contracts
- Supported 13 graduate research assistants through the PREP, the NIST foreign guest researcher program, and the NSF Math Science Internship program
- Supported the part time work of 8 faculty members
- Formally engaged with many others as unpaid guest researchers
- Informally engaged with many additional collaborators worldwide.
Recommendation: The [Computer Security Division] (CSD) should consider adding staff to the Lightweight Cryptography project.
ITL response: Complete: <3 new staff members were added> to the Lightweight Cryptography project.
Recommendation: The CSD should consider adding staff to the Combinatorial Methods in Software Testing project to accelerate adoption of the project’s tools and techniques by the software development community.
ITL response: Complete: <A staff member> was added to Combinatorial Methods team.
Recommendation: The CSD should devote additional short-term resources to Common Vulnerabilities and Exposures [CVE] scoring until the backlog can be remediated.
ITL response: Complete: CVE backlog issue from that period addressed. Current and new CVE issue being addressed with new short-term resource allocations.
Recommendation: The CSD should emphasize recruiting of mid-career staff.
ITL response: Complete and Ongoing: New mid-career staff added in each group.
Recommendation: The ITL should expedite and grow the Professional Research Experience Program [PREP] to hire more international graduate students from among those already at U.S. universities (e.g., as interns or cooperative researchers).
ITL response: ITL continues to expand our use of the PREP program—since the last panel meeting ITL has employed 42 new PREP staff, including international students attending U.S. universities.
Recommendation: The ITL should assess the effectiveness of its efforts to improve recruiting, retention, and mentoring of women and minorities.
ITL response: According to NIST HR DATA, ITL’s candidate pools between May 2021–Feb 2024 for all pay plans (ZA, ZS, ZP) was 48% minority, 32% nonminority, 20% omitted. During the same period ITL’s candidate pools were 35% female, 47% male, 19% omitted. During this period the percent of total staff increased by 4% for females, 3% for minorities, and our hires in this period were 50% females. ITL leadership emphasizes the importance of seeking diverse candidate pools—and the use of tools such as LinkedIn recruiter. The staff led ITL Diversity Committee is implementing its strategic plan in coordination with management. ITL proposed tools that NIST now uses to ensure language of job openings is not biased. All Group Leaders are being interviewed about recruiting and retention to share best practices and lessons learned.
Technical Planning
Recommendation: The ACMD should engage in a formal strategic planning exercise with the following goals:
- Identify current core competencies and match them to NIST needs;
- Identify gaps and new opportunities—mapping what its strategic goals are to resources (budget and staff)—in emerging areas such as artificial intelligence and machine learning; and
- Engage the next generation of ACMD leaders in developing this plan, so that what emerges can be enthusiastically executed by them.
ITL Response: An Applied and Computational Mathematics Division Capability Plan was developed in 2019 which has the requested features.
Table of Contents
1 Introduction: The Division and Its
Operations
Customers
Approach
Relation to Internal Customers
Relation to External Customers
Project Selection
2 Capabilities Needed for the Future
2.1 Math and Comp Foundations of Adv Metrology
2.2 Future Computing Technologies
2.3 Mathematical Knowledge Management
3 Meta Issues
3.1 Coping with a Wave of Retirements
3.2 Ensuring Diversity in the Workforce
3.3 Developing Competencies in New Areas
3.4 Developing the NIST Customer Base
3.5 Physical Location of NIST Staff
4 Staffing Trends and Needs
Technical Areas Identified for Growth
- Quantum-based measurements
- Bioscience*
- Measurement science for information technology
- Data, machine learning, and AI*
- Dynamic metrology
- Imaging systems as metrological devices
- Multiscale material modeling
- Metrology for modeling and simulation
- Quantum information theory*
- Quantum architectures, benchmarking, and testing*
- Quantum communication systems and components*
- Neuromorphic computing
- Mathematical knowledge management
While there has been activity in all these areas, those asterisked have seen the biggest increases.
Conferences and Publications
Recommendation: The ITL should perform a systematic assessment of the conferences at which its staff members have presented their research or otherwise attended. The ITL should consider whether attendance has been sufficiently frequent and whether the conferences are of sufficiently high quality, and it should maintain or increase, as appropriate, conference attendance. A similar assessment should be performed for publications in scholarly journals.
ITL Response: Conference attendance is determined by Division management who assess the return on investment with respect to advancing the NIST mission in deciding when to send staff to conferences. Presenting research results is the highest priority, but the importance of staff development by engaging with national and international collaborators on the latest research, is another important consideration. ITL publishes in both conference proceedings and scholarly journals—depending on the type of research and the target audience.
Recommendation: The ACMD should evaluate simulation software development practices in light of the disruptive changes in high-performance computing technology.
ITL Response: Individual staff members have continued to engage in self-study to increase knowledge, skills, and abilities in this area. Machine learning techniques and workflows is one example. Research software engineering is an area in which we would like to grow, and we have had some recent contract support in this area, but budgetary and recruiting considerations make expansion a challenge.
Recommendation: The Access Control project’s resources should be directed toward more recently emergent risks in order to have higher impact.
ITL Response: Complete: Access control shifted to tech transfer opportunities and newer technologies.
Recommendation: The CSD should take steps to publicize the Lightweight Cryptography [LWC] program among potential users of the resulting algorithms—particularly Internet of Things [IOT] vendors and customers.
ITL Response: Complete and Ongoing: LWC project continues as final algorithm selection is finished. Direct outreach includes to the IOT program and potential affected communities (i.e., space, automotive, etc.).
Recommendation: Recognizing that impact is sometimes difficult to measure without deep insight into stakeholder products and processes, the ITL should work toward the development of impact metrics for projects in the CSD where development of such metrics is feasible.
ITL Response: Complete and Ongoing: Crosses several avenues from annual reviews of use of posted references, completion of impact studies, reviews of industry access to CSD data and [Standards Developing Organization] adoption.
Recommendation: The CSD, in partnership with the [Applied Cybersecurity Division] ACD, should investigate and, if possible, develop and disseminate metrics for privacy.
ITL Response:
- CSD: Completion of Privacy Controls in SP 800-53 and Privacy Assessment Methods in SP 800-53A.
- ACD: The Privacy Engineering Program established the Collaborative Research Cycle to benchmark data de-identification techniques and develop metrics.
Recommendation: The ITL should consider putting together a rapid response plan of action to be invoked in the event of a real-world safety or security problem after a technology has adhered to the best practices and guidance from the [National Cybersecurity Center of Excellence] NCCoE. To the extent that there is the potential for reputational damage to NIST as to the effectiveness of its best practices and guidance, the ACD should prepare in advance to proactively address issues that may arise.
ITL Response: The recommendation was elevated to ITL for crisis communication preparation, including tabletops, for all NIST cybersecurity publications.
Recommendation: The NCCoE should add an adversarial perspective to the solutions and guidance that are promulgated by the NCCoE laboratories. That would mean conducting an adversarial review (e.g., red-teaming) against these solutions and feeding the adversarial review results back into their process for purposes of defensive improvement. This may involve adding steps into the current NCCoE process before reference designs and documents are released from the laboratory; additional resources should be added if needed to accomplish including the additional steps.
ITL Response: Recommendation not implemented. The NCCoE reviewed the recommendation and considered the value that adversarial review (i.e., red-teaming) would bring to a project solution. The value of NCCoE technical projects is the architecture with the products involved to provide examples that demonstrate possible solutions. An adversarial red-team type of review would test the products within a build rather than the architecture. While this could provide value to a specific lab instance, it would be detrimental to the NCCoE’s relationship with its collaborators. It would also only be relevant for organizations using the exact setup and products indicated in the NCCoE project. In addition, NCCoE
builds only represent the part of an architecture that is relevant to the security challenge in question. It is likely that organizations will have additional controls in place that would not be tested in this type of red team activity.
Recommendation: The NCCoE should examine the university affiliates program with the federally funded research and development center contractor and consider how that program could be modified to enhance engagement with the existing university affiliates and how it could be improved to broaden participation with additional universities.
ITL Response: NCCoE sought to participate with universities through 1. Capstones for students to research a topic of interest to the NCCoE and 2. Expansion of student work-based learning opportunities that leverage both NIST and MITRE intern programs. New NCCoE leadership is exploring how to increase stakeholder engagement (including academia) on NCCoE projects, including to define cybersecurity challenges and increase use of NCCoE outputs. NCCoE hopes to find ways to leverage the academic community for new project ideas, as well as technical participation on projects.
Recommendation: The NCCoE should develop a process by which results from the field are systematically and proactively tracked and monitored after a project has been successfully transferred out of the NCCoE laboratory. The results from this proactive monitoring should then be disseminated (e.g., by the NIST Special Publications 1800 series) and appropriately incorporated into future NCCoE laboratory projects.
ITL Response: The NCCoE tracks several metrics associated with impact, including publication downloads, event attendance, COI subscribers, CRADA numbers, as well as qualitative discussions with the community on the use of NCCoE outputs.
New leadership has defined the need to measure impact of the NCCoE’s work as a top priority. Tiger teams have been recently stood up representing leadership and engineers across NIST and MITRE to identify additional metrics to define impact, set up a process for project reviews, as well as find ways to increase communication on successes across the NCCoE and with the public.
2021 REPORT
Technical Expertise of the Staff and Adequacy of Staffing
Recommendation: ITL should apply an aggressive, imaginative focus on hiring to replace retiring staff, to address important growth areas such as artificial intelligence, machine learning, and data science, and to fill specific gaps in the divisions. This effort should aspire to diversity targets.
ITL Response: ITL has hired 17 new staff in [artificial intelligence], [machine learning], and data science across all our divisions. When recruiting we ignite candidates’ imaginations regarding how they can work in ITL to solve national and international problems—e.g., the safe and trustworthy use of Artificial Intelligence, cybersecurity, privacy, quantum computing and networking and much more. We receive large number of candidates for these job openings.
Recommendation: ITL should plan and implement effective ways to recruit and retain a diverse workforce to ensure the appropriate staffing in areas of significant interest to national welfare and security, and to address severe competition from industry in areas such as artificial intelligence, cybersecurity, and the [IOT].
ITL Response: ITL’s diversity committee developed a strategic plan with 4 main objectives, 26 strategies, and associated success measures to improve recruiting and retention of a diverse
workforce. ITL implemented a Speaker’s Bureau and reached out to minority serving universities to offer ITL experts to give talks about their work and opportunities at NIST. ITL recruits staff who are motivated to make a difference in the world through our research, standards, measurements, testing, and guidance in critical areas such as [artificial intelligence], cybersecurity, privacy, quantum, biometrics, software testing, and others. ITL also uses all HR capabilities to retain staff, such as retention bonuses.
Recommendation: ITL should establish exchange programs with relevant government laboratories, academic institutions, and industry consortia to stimulate new ideas and problem areas, enhance competencies, and facilitate collaboration.
ITL Response: ITL actively pursues staff exchange—with researchers coming to NIST as well as NIST staff going to other organizations. Since our last panel meeting ITL has had 59 new staff exchanges with universities and government agencies both in the U.S. and around the world.
Adequacy of Facilities and Equipment
Recommendation: ITL should take steps to ensure adequate resources, especially computing to support AI/ML and data science at sufficient scale.
ITL Response: ITL provided 3 computing experts to the NIST Research Computing Infrastructure Task Force. With ITL leadership the task force succeeded in getting approval for a complete renovation and update of NIST’s local computing resources as well as formal plans for access to external HPC resources. The Task Force developed a detailed and compelling vision for how access to adequate computing resources is critical to NIST’s future. This investment will benefit ITL, and all NIST researchers, through an effective, scalable, shared approach to computing infrastructure.
Recommendation. To get access to the most modern resources, ITL should seek collaborations with other organizations in the public and private sectors, including other Government agencies. To achieve collaborative access, the ITL should examine its potential contributions to partnerships.
ITL Response: Since our last [National Academies] review, ITL has established 469 new collaborations to expand access to new methodologies and approaches, and access modern resources. These collaborations are critical to all the work carried out in ITL—nearly all ITL staff have external collaborators for their work.
Effectiveness of the Dissemination of Outputs
Recommendation: ITL should broaden its impact to non-technical stakeholders, policy makers, and the public.
ITL Response: ITL communicates its value through the use of the internet, including social media (e.g., @NISTCyber) and high visibility events with Congress, the White House, workshops, and conferences. In addition, ITL actively supports Take Your Kids to Work Day to encourage youth participation in STEM. More information on the dissemination of outputs and technology transfer for the three divisions under review is available in the read-ahead material and on the ITL NASEM supplemental webpage and in the infographic below [Figure 6-1].
SOURCE: Courtesy of NIST Information Technology Laboratory.
REFERENCES
NASEM (National Academies of Sciences, Engineering, and Medicine). 2018. An Assessment of Four Divisions of the Information Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2018. The National Academies Press. https://doi.org/10.17226/25283.
NASEM. 2021. An Assessment of Selected Divisions of the Information Technology Laboratory at the National Institute of Standards and Technology: Fiscal Year 2021. The National Academies Press. https://doi.org/10.17226/26354.
