An Assessment of Selected Research Programs and Goals of the Engineering Laboratory at the National Institute of Standards and Technology: Fiscal Year 2024 (2025)
Chapter: 7 Overarching Themes, Key Recommendations, and Chapter-Specific Recommendations
7
Overarching Themes, Key Recommendations, and Chapter-Specific Recommendations
OVERARCHING THEMES AND KEY RECOMMENDATIONS
In the course of conducting its work—reviewing printed materials, receiving presentations, taking tours, and talking with Engineering Laboratory (EL) staff—some common themes appeared across the different subpanels and their chapters. Because the subpanels worked mostly independently of each other, this is indicative of situations that pertain to the institution being assessed as a whole rather than something that only pertains to a part of its work. These are overarching themes and the recommendations to address them are key recommendations.
In this assessment, the overarching themes that emerged were (1) strategic direction and strategic planning and (2) metrics to help determine when a project has been successful, to determine when it is time to sunset a project to free up resources for new efforts, and to ensure that the work is relevant to the stakeholders.
Strategic Direction and Strategic Planning
It was not clear to the panel how the strategies and strategic directions of EL’s work are set. To understand and assess the effectiveness of programs at EL, the panel needed to understand the goals and objectives of EL. How does EL know that the program areas in which its researchers are working align with the mission of the laboratory, the National Institute of Standards and Technology (NIST) more broadly, and stakeholder needs? The panel was not provided with a strategic plan, a short- or long-term implementation plan, key performance indicators, or reporting requirements.
Without strategic guidance from clearly set goals, it is very difficult to evaluate whether the necessary expertise, budget, and facilities are in place to sustainably support globally competitive programs in EL’s areas of interest. An important concern is a perception that EL does things because they can be done, not that they pursue work directed as meeting a defined set of needs. A useful strategic planning process will make it clear why the programs are doing what they are doing. It will establish clear benchmarks through which to judge progress and, in turn, establish clear go/no-go decision points. When projects are being selected, it would be useful to have an EL management review to look across the laboratory, NIST, and the broader stakeholder community and ensure that there are no duplications of effort and to help identify opportunities for collaboration.
Many of the research programs presented to the panel seem to operate in a stand-alone and non-interactive way. EL may consider ways to combine and consolidate research projects that can benefit from collective efforts, thereby enhancing overall efficiency. Some of the appearance of stand-alone and non-interactive programs may be a consequence of individual projects being assigned to one principal investigator that focuses on a single project and undertakes all the work of the project themselves. When principal investigators are funded individually for their research annually, rather than EL being funded based on strategic initiatives, it becomes difficult to foster a collaborative environment that promotes
program success. Examples of things that could help to ensure that EL is applying its limited resources most effectively are conference attendance, hosting onsite and offsite workshops, and hosting activities that include a mix of industry and research entities. Strategic planning would also guide staff recruitment with an eye to the big picture.
To ensure maximum impact, it is desirable that EL’s strategic directions and planning align with needs of its stakeholders, including industry. It is unclear how EL determines that its work is best aligned with stakeholder needs. Without this strategic alignment, it is difficult to determine if EL is providing the most relevant, cost-effective, and meaningful contributions to its stakeholders. Engaging with relevant industry and other stakeholder groups would facilitate a deeper understanding of their needs, enabling targeted project development that directly addresses stakeholder challenges and priorities. This approach would enhance the relevance and effectiveness of projects, fostering meaningful contributions to advancements and innovation in industry and other stakeholder communities.
Key Recommendation 1: The Engineering Laboratory (EL) should determine and describe the strategic directions it will pursue and how personnel and resources will be allocated to pursue those directions. Management reviews during project selection should use the strategic directions to avoid duplication, look for collaboration opportunities, and guide the recruiting of any new staff. EL should also develop and publish a long-term strategic plan that includes input from stakeholders such as industry.
Metrics and Stakeholder Relevance
While its mission states that EL “promotes U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology for engineered systems in ways that enhance economic security and improve quality of life,” the question left with the panel is “How?” How does EL know that it is meeting its mission, and what are the metrics used to determine success? Another need is for evaluation criteria and metrics to be used to select and then judge the success of projects. It was not obvious to the panel that there are criteria for evaluating how programs are selected, nor are there metrics to measure the success of programs or that could be used in deciding whether to maintain or terminate a program of research. It was also not clear if programs are evaluated periodically for validity or stakeholder relevance. The panel was unable to acquire granular information on project benchmarking and the measurement of key performance on projects. Individual measurements of effectiveness and their impact need to be done at the individual project level in EL. Metrics do not have to be perfect to be useful. Some suggested metrics (some of which are in place) may include things such as achieving project milestones on schedule, on-time completion of work, software downloads, publications, citations, industry adoption of software, standards integration, report generation, patents and licensing, and recognition through awards, among others.
Presuming that EL is already periodically evaluating the performance and program success metrics, it would be useful for EL to communicate its track record and success with past and future programs and value, allowing for the consideration of the diverse range of outputs from EL’s work. NIST aims to become a world-class organization, and it may benefit from further efforts to define the meaning of “world class”—such a definition would assist assessment panels in providing feedback on achieving that goal.
The pace of work also impacts industry relevance. Research projects typically follow 2–5-year cycles, which is common in industry. However, due to rapid technological advancements, long-term research projects risk producing outdated results by the time they are completed. To meet industry needs in particular, EL might consider reassessing project milestones every 12 months. This would provide an opportunity to adjust milestones as needed to ensure that the work remains relevant and impactful.
In general, EL appears to not be externally communicating success metrics that demonstrate how its research is impactful. As shown in Figure 2-5, the budgets for EL’s goals have either remained stable, increased slightly over time, or decreased (the decrease in Figure 2-5 was due to a reorganization with
some of EL’s groups moving to the Communications Technology Laboratory). These data, however, are not corrected for inflation. This means that EL has likely either been barely maintaining purchasing power in the face of inflation or losing purchasing power. Clear metrics of the sort recommended here, and the clear communication of them to stakeholders and funders, could also be used to demonstrate EL’s value proposition to appropriators.
Key Recommendation 2: The Engineering Laboratory (EL) should communicate the current clear metrics for project adoption and success that are already in place. A management review of the work portfolio should be considered as a tool to determine whether to keep a project going or terminate it. EL should communicate the metrics externally to stakeholders and review panels. The adoption of standards by industrial groups to which EL has contributed should be externally communicated to stakeholders regularly. EL should also define what it means to be “world-class” so it can know when it is meeting this goal.
CHAPTER-SPECIFIC RECOMMENDATIONS
Advanced Manufacturing Goal: Measurement Science for Additive Manufacturing Program
Recommendation 3-1: The Measurement Science for Additive Manufacturing Program should expand the testbed concept to include data-related projects and to support tighter integration between data-related projects and hardware-related projects.
Recommendation 3-2: The Measurement Science for Additive Manufacturing (AM) Program’s leadership should refine the goals of the Fundamental Measurements for Metals AM, Advanced Informatics and Artificial Intelligence for AM, and Data Management and Fusion for AM Industrialization projects to be more focused and measurable, integrated with experimental practice, and aligned with realistically available resources. The program’s leadership should clearly define its role in the wider AM space and explore collaborations to leverage and achieve a maximum return on the investment of its resources.
Recommendation 3-3: The Measurement Science for Additive Manufacturing program should better align team member expertise with practical, industrial measurement science needs. This can be accomplished through upskilling in applied data science and artificial intelligence. Also, they should add experts with relevant skills, such as powder rheology, the state of the art in powder testing and modeling techniques, and qualification methods for machines and processes that use powder feedstock. Finally, more direct engagement with industry would greatly help.
Recommendation 3-4: The Measurement Science for Additive Manufacturing program should explore other ways to disseminate the results of its work directly to industry. These could include mailing lists to inform industry stakeholders of new publications in a timely manner, producing more National Institute of Standards and Technology reports in areas that are industry-relevant but less likely to result in peer-reviewed papers, and making as many peer-reviewed papers openly accessible as possible.
Advanced Manufacturing Goal: Measurement Science for Manufacturing Robotics Program
Recommendation 4-1: The Measurement Science for Manufacturing Robotics Program should discontinue the use of the term “cobot” and align with International Organization for Standardization Standard 10218 and ANSI/RIA Standard R15.06.
Recommendation 4-2: The Perception Performance of Robotic Systems project should develop an intentional work statement focused on vision-based tracking systems with clear connections into adjacent projects addressing human–robot interactions and mobility and exoskeleton projects.
Recommendation 4-3: The Performance of Human–Robot Interaction (HRI) project should explore emerging human-centric cognitive and physical standards such as Institute of Electrical and Electronics Engineers P7017, Recommended Practice for Design-Centered Human–Robot Interaction and Governance. The data and models from this project in the National Institute of Standards and Technology repository should be disseminated to researchers employing artificial intelligence and machine learning to help drive future research activities and collaborations. If this project explores human cognition within HRI, it should obtain cognitive architecture capabilities as well as associated high-performance computational hardware.
Recommendation 4-4: The Embodied Artificial Intelligence (AI) and Data Generation for Manufacturing Robots project should develop and communicate clear roadmaps to achieve their stated research objectives that characterize robotic systems that involve AI algorithms, training paradigms, and metrics.
Recommendation 4-5: The Measurement Science for Manufacturing Robotics Program (MSMR) should partner with universities and other research institutions with expertise in the rapidly advancing technologies relevant to its work, such as artificial intelligence, cognitive science, and human-robot interaction technologies. This would help MSMR to maintain its relevance to, and maximize its impact on, industry and U.S. competitiveness.
Recommendation 4-6: All projects in the Measurement Science for Manufacturing Robotics Program should expand their engagement with manufacturing innovation institutes to bolster dissemination and engagement with large and small system integrators and end users.
Advanced Manufacturing Goal: Advanced Manufacturing Data Infrastructure and Analytics Program
Recommendation 5-1: The Engineering Laboratory should increase internal collaboration across different programs within it to leverage existing expertise and resources to maximum effect.
Recommendation 5-2: The Engineering Laboratory should assess the obstacles in current industrial practices preventing the advancement of a data infrastructure that will improve productivity, resiliency, and sustainability for manufacturing operations and supply chains, and identify targeted research programs to overcome these obstacles.
Recommendation 5-3: The Engineering Laboratory should create yearly a prioritized list of industry challenges and develop corresponding National Institute of Standards and Technology problem statements that align with those industrial needs. The list and problem statements should specify goals and include a detailed schedule and specify the deliverables that are to be disseminated. All of this should be used to obtain the resources—both people and funding—to achieve the goals in the problem statements.
Recommendation 5-4: The Engineering Laboratory (EL) should conduct a clear needs assessment to understand industry’s needs and how EL can best address them, integrating its efforts into the broader efforts under way in industry. A timeline to address those needs, reflecting the speed at which things move in industry, should also be developed.
Recommendation 5-5: The Augmented Intelligence for Manufacturing Systems Project should develop strong collaborations with established industry consortia and institutes such as those comprised under Manufacturing USA.
Recommendation 5-6: The Engineering Laboratory should make a plan to transition the results of its measurement science work to industry to ensure that its work is relevant to industry’s needs.
Energy-Efficient, High-Performance Buildings Goal: Net-Zero Energy, High-Performance Buildings and Embedded Intelligence in Buildings Programs
Recommendation 6-1: The Engineering Laboratory should foster collaboration with other laboratories specializing in artificial intelligence and machine learning to leverage their expertise. The Information Technology Laboratory would be a good place to start.
Recommendation 6-2: The leadership of the Energy-Efficient, High-Performance Buildings Goal should clearly communicate their facility shortcomings and needs to the Office of Facilities and Property Management so those needs can be reflected in annual budget requests and in facility master planning activities.
Recommendation 6-3: The Engineering Laboratory (EL) should develop and implement a space management plan that ties in with the strategic plan proposed in Key Recommendation 1. EL should use scheduling software where research laboratories can create central platforms to manage and monitor the availability and use of spaces and equipment.
Recommendation 6-4: The Engineering Laboratory leadership needs to develop and implement a long-term staffing plan to identify leaders and develop people in key areas. This plan should include elements such as the transfer of knowledge and experience from older workers to younger ones, formalized mentoring arrangements, and finding opportunities for EL researchers who show promise to work in other laboratories and divisions across the whole of the National Institute of Standards and Technology to develop them professionally.
Recommendation 6-5: The leadership in the Energy-Efficient, High-Performance Buildings Goal area should define succinct metrics to measure dissemination. They should look for ways to go beyond counting numbers and look for ways to judge the impact of this goal’s work products.
