An Assessment of Selected Research Programs and Goals of the Engineering Laboratory at the National Institute of Standards and Technology: Fiscal Year 2024 (2025)
Chapter: 4 Advanced Manufacturing Goal: Measurement Science for Manufacturing Robotics Program
4
Advanced Manufacturing Goal: Measurement Science for Manufacturing Robotics Program
Domestic manufacturing is essential to U.S. national security and economic competitiveness. In March 2024, there were 570,000 manufacturing job openings, and 3.8 million new positions are expected by 2033 (DeLoitte 2024; NAM 2024). Robotic manufacturing addresses critical workforce shortages, especially in areas needing highly skilled, experienced workers. Robotics offers the potential to reduce the proportion of “dull, dirty, and dangerous” jobs, so that the remaining positions are safer, more interesting, and more rewarding to attract and retain new talent in the manufacturing sector.
The Measurement Science for Manufacturing Robotics (MSMR) program develops and deploys measurement science that advances manufacturing robotic system performance, collaboration, agility, autonomy, safety, and ease of implementation to enhance U.S. innovation and industrial competitiveness. The work of this program is designed to support the creation of better products, manufacturing processes, and, importantly, new and improved standards for manufacturing robotics. Emerging technologies such as artificial intelligence (AI), machine learning, autonomy, cognitive science, and human–robot interactions (HRIs) are already impacting the development of new robotic manufacturing capabilities. The MSMR program needs to maintain its high standard of technical leadership to keep pace with industry and other stakeholder needs, and to disseminate its contributions in a manner accessible to small- and medium-sized manufacturers that contribute a large fraction of U.S. production as well as the larger manufacturers.
The MSMR program is organized into the following eight projects:
- Grasping, Manipulation, and Contact Safety Performance of Robotic Systems
- Perception Performance of Robotic Systems
- Agility Performance of Robotic Systems
- Mobility Performance of Robotic Systems (Exosuits)
- Performance of Human–Robot Interaction
- Emergency Response Robots
- Embodied AI and Data Generation for Manufacturing Robotics
- Digital Twins and Emerging Technologies for Small-and-Medium Enterprise (SME) Work Cells
GENERAL OBSERVATIONS
MSMR has strong technical competence in measurement science as applied to agility, grasping and manipulation, mobility, exosuits, perception, HRI, and emergency response. Equipment and facilities are sufficient to meet the project objectives and are in some cases state of the art compared to leading universities and research institutions. The program’s staff have strong representation and leadership in working groups at ASTM International and other standards-setting organizations. The experimental, theoretical, and data analytics results of the program’s projects support technical leadership in the working groups. Information that supports standards is disseminated effectively to research communities through publications and presentations in leading refereed journals and conferences.
MSMR addresses the need in the external research community for the objective testing of competing technical solutions for manufacturing-relevant problems. The strongest example is the dissemination of Assembly Task Boards (ATBs). ATBs present manipulation challenges that go beyond the state of the art, commercially available technology and are used widely in competitions at leading international conferences, encouraging collaboration on multi-disciplinary teams.
The information MSMR presented to the panel was insufficient to assess the quality and strength of a strategic plan and technical roadmap on how (1) each task in the plan contributed directly to clearly defined goals, (2) integration with technologies and results from external sources, (3) gaps not addressed, and (4) mechanisms for engagement and transition to end users in industry.
There were also no clear metrics that quantify and track the success and impact of MSMR’s work and whether its work is on target with its mission. Engineering Laboratory (EL) programs frequently reference benefits to SMEs without quantifying operational productivity, quality, or financial impacts, nor mechanisms for collecting and tracking the data. Such metrics have value in setting priorities, resource allocation, and accountability for results, and enhance credibility and engagement with industry and other stakeholders. Strategic planning and metrics are discussed in more detail in Chapter 7.
ASSESSMENT OF TECHNICAL PROGRAMS
Accomplishments
Grasping, Manipulation, and Contact Safety Performance of Robotic Systems
Most manufacturing manual labor requires the grasping and manipulation of mechanical objects such as raw materials, workpieces, tooling, and fixtures. End-of-arm tooling grippers enable the automation of many manufacturing processes, with leading suppliers offering a wide variety of gripper geometries and actuation mechanisms to meet the requirements of specific applications. The Grasping, Manipulation, and Contact Safety Performance of Robotic Systems project develops and standardizes performance metrics, test methods, and associated measurement tools that support the development of robotic systems that have human-like dexterity and force control characteristics.
Mechanical grippers are widely available based on mature technologies for rigid objects for a wide range of geometries. Data from leading gripper suppliers are limited to the specification of gripping force, actuation time, finger strength, and repeat accuracy, with no reference to standards or test methods used. This project continues to play an important supporting role in developing standards for performance metrics, test methods, and measurement tools for grasping and manipulation, addressing industry needs for robot system integrators and end users. Specific contributions include participation on the ASTM International subcommittee F45.05 on Grasping and Manipulation and technical publications for ASTM Working Groups WK83863 and 86189 covering grasp strength performance and measurement apparatus. Test protocols included pinching and wrapping types for small, medium, and large workpiece artifacts, with methods for calculating statistical confidence limits.
Compared to rigid body grippers, technologies supporting human-like dexterity are much less mature. Assembly tasks for rigid parts such as gears and threads require more precise control and touch than basic gripping. Tasks for assembling belt drives and wire harnesses are more challenging due to variations in their stiffness and changes in shape as the products move from their initial to final states. The research community benefits from unified sets of benchmark tasks and results for reliable and repeatable comparisons. Working through ASTM Working Groups WK87214 and WK82713, the EL staff working on this project has developed National Institute of Standards and Technology (NIST) ATBs. ATBs are supported by protocols for use, time-based comparisons to human performance, and support for competitions globally such as the Institute of Electrical and Electronics Engineers (IEEE) International Conference on Robotics and Automation Competitions in May 2024. Advancements in the manipulation
of flexible materials such as textiles and ceramic composites will benefit the aerospace, personal protective equipment, and fashion segments.
A more recent initiative addresses the development of new safety standards for the impact on humans from power, force, and torque-limited (PFL) robots1 that could induce pain, assisting in the development of International Organization for Standardization (ISO) Technical Specification 15066 and ISO Publicly Available Specification 5672:2023. Building on the EL tradition of innovation in measurement, the project staff has developed a soft pressure sensor that matches the biomedical response of the human forearm, using a linear change in capacitance with applied force.
Perception Performance of Robotic Systems
Using robotics in industrial applications frequently involves sensors and related data analysis to enable perception capabilities. The types of sensors (and ancillary devices like lighting) have a range of use cases and means of capturing and generating data. Equally important is how the data are analyzed and resolved to be used by robotic systems. This foundational capability set affects other topics like mobility, grasping and manipulation, and HRI, among others.
The project’s five key areas of research emphasis address a range of topics and integrate well with adjacent projects like Grasping, Manipulation, and Contact Safety Performance of Robotic Systems and Performance of HRI. The first of the five, bin-picking, encompasses a deceptively broad array of perception issues and is of growing industry use and importance making it a reasonable use case on which to base project deliverables. Other areas—three-dimensional (3D) vision under variable lighting, resolving geometric features, estimating atypical part poses, and 3D vision selection criteria—all logically relate to and complement one another. This project has good evidence of engagement with industry through participation at relevant industry events like the Association for Advancing Automation (A3) Automate and The Vision Show, as well as co-hosting a workshop with ASTM’s Committee E57 to uncover challenges and solutions related to 3D perception systems for robotics and to develop a roadmap of highest priority needed standards to work on most urgently. There is a clear focus on translating testing research into standards collaboratively with ASTM’s subcommittee E57.23 leading to four new working groups. This resulted in the publication and adoption by industry of the ASTM E57 standard for point cloud data formatting.
Agility Performance of Robotic Systems
This project aims to develop measurement science to evaluate and ensure the agility of robotic systems. This includes metrics, test methods, information models, and planning approaches, validated in a combined virtual and real testing environment. This will enable manufacturers to reconfigure and re-task robots more quickly, making them more accessible to smaller organizations and increasing efficiency in larger ones.
A major accomplishment of this program is establishing the Agile Robotics for Industrial Automation Competition (ARIAC), which has been running since 2018. ARIAC is a yearly contest, involving over 50 competitors, that evaluates robots’ adaptability, efficiency, and autonomy in performing tasks in a simulated manufacturing facility. The competition aids in the development of algorithms for real-world manufacturing, and EL uses the findings to establish standard metrics and test methods for future robotic agility. The participants in ARIAC have typically been students and postdoctoral researchers. In addition, the team members have been engaged in four IEEE standards working groups. The project team also presented a framework for agility performance evaluation that it is developing. The framework included elements of hardware, software, and communication reconfigurability (agility) that considered robot sensing, perception, reasoning, task representation, task planning, and execution.
___________________
1 Power, force, and torque-limited robots were called “cobots” when first introduced. This term is now obsolete, replaced by safety definitions in ISO Standard 10218/R15.06.
Mobility Performance of Robotic Systems (Exosuits)
This project’s research is focused on (1) the dynamic localization of autonomous guided vehicles (A-UGVs) and mobile manipulators operating with no human guidance, in relation to tooling and workpiece targets while considering both continuous and non-continuous motion and (2) developing metrics and test methods for evaluating the performance of exoskeletons for industrial applications. The work is leading edge and relevant to industry, as is the evaluation of localization as a function of uncertainty in workpiece location and other attributes, a common real-world experience. Incorporating additional sensor suites (e.g., light acuity and optical tracking) on top of industrially relevant components (e.g., Omron Autonomous Mobile Robot and Universal Robots manipulator) is a good way to facilitate broader test conditions and establish ground truth.
There is a strong rationale to support this project’s research direction, related to wearables and international coordination with a reasonably wide segment of stakeholders, even if it is somewhat skewed to the academic side of things.
Performance of Human–Robot Interaction
Robots deployed in manufacturing and industrial environments tend to be prohibited from being in the immediate working area of their human operators. To increase production throughput dramatically, collaborative and more direct interactions between humans and robots are necessary to minimize process variation and enhance operational efficiencies. This project employs measurement science to assess and assure the usability, performance, and trustworthiness of HRIs in manufacturing environments. Target outputs will assist system integrators and manufacturer end users, especially in high-mix, low-volume operations.
This project’s laboratory provides the user community with a glimpse of what peer-like interactions are possible between a robot and a human. The results will lead to future trust-based models and explorations into ethics. Project staff appear to reflect a solid engineering science knowledge base, with a great set of skills and understanding the current state of the art in human–machine interfaces.
This project’s staff team is on par in conference and journal publications compared to peer laboratories, such as universities in the same field. The laboratory participates in and leads various Association for Computing Machinery (ACM) and IEEE entities to generate emerging HRI standards for use in industry. The project’s facility contains state-of-the-art equipment and computational hardware that is on par with other research facilities and is adequate to achieve the stated research objectives. Besides robots, sensors, and commercially available augmented reality and virtual reality devices, the laboratory has a motion capture and rigid body tracking system to enable the initial exploration of collaborative interactions beyond existing techniques based on force and torque. All equipment as well as the facility are relevant and up to date.
Emergency Response Robots Project
Over the past few decades, robots have evolved into the tools of choice for first responders in extremely hazardous scenarios. This project employs the use of measurement science to evaluate the capabilities of autonomous and remotely operated robotic systems—ground, aerial, and aquatic—for emergency response applications. This project also examines the training and resultant proficiency of remote operators in a variety of emergency response applications.
This project has developed more than 50 test procedures with 25 of these procedures leading to ASTM standards. Recently, the laboratory has convened or participated in challenges involving quadruped platform mobility through various terrains and obstacles, which has led to additional standardized test procedures. For ground robotics, standards were published for terminology (e.g., definition of a ground robot), logistics, maneuver, terrain mobility, confined mobility with obstacles, dexterity, situational awareness, sensing, endurance, and radio communications. For aerial drones, a standard was published for endurance, with another standard currently in draft for remote operatory
proficiency. Finally, for aquatic robots, various test methods are in draft form. Besides the numerous awards over the past decade, such as the Department of Homeland Security Secretary’s Award for Excellence in Innovation, the project has supported more than $200 million in data-driven commercial procurements, which shows project impact by assisting in the user acquisition of such systems.
Embodied Artificial Intelligence and Data Generation for Manufacturing Robotics
The objective of this project is to facilitate the adoption of AI-based robotic approaches in practical manufacturing scenarios by creating test methods that target AI-enabled robotic systems, evaluating the performance of AI-enabled robotic systems, and creating manufacturing-relevant and AI-centric data sets. The technical idea is to provide manufacturers and integrators with a streamlined way of assessing the productive impact of AI systems, which NIST can provide through AI-specific productivity metrics and test methods.
The project developed an advanced six-degree-of-freedom sensor to support the use of PFL robots in high-accuracy applications. PFL robots face rigidity and accuracy challenges compared to traditional industrial robots. The sensor is novel using line features instead of point features to create a coordinate system. The project was presented at the IEEE 2023 International Conference on Automation Science and Engineering and received U.S. patent 10885368.
The Manufacturing Objects for Assembly Data Set (MOAD) effort is developing two-dimensional and 3D visual sensor data to be used in support of robot system solutions. The database provides high-resolution red-green-blue depth scans, physical properties, and geometric models of objects for easy incorporation into manipulation and planning software platforms. This effort demonstrates excellent collaboration with the Grasping, Manipulation, and Contact Safety Performance of Robotic Systems team.
Digital Twins and Emerging Technologies for Small-and-Medium Enterprise Work Cells
This project aims to identify automation technologies suitable for SMEs, develop tools to facilitate their integration, and address the need for robots to operate in unstructured environments. It will leverage emerging technologies like digital twins to ease the selection and installation of robotic systems, enhance operational robustness, and enable seamless integration of sensors and other tools. The digital twins will encompass software, models, prototype artifacts, test methods, metrics, and standards that facilitate the integration and evaluation of emerging technologies in SME environments. These outputs will use sensor-derived information about the workspace to address situations in which robots must perform tasks in less-structured surroundings. This program will also serve as a bridge for interactions with small manufacturers, systems integrators, the NIST Manufacturing Extension Partnership (MEP) centers, and other stakeholders to identify the major obstacles to greater adoption of robots and the ongoing needs of SMEs when using manufacturing robots.
One of the program’s accomplishments was an extensive 2023 survey of SMEs querying their current levels of automation, providing insights into small manufacturing in the United States. In-depth analysis of the results of this survey is ongoing. All 51 MEPs were used as a mechanism to reach SMEs across the United States.
Opportunities and Challenges
Grasping, Manipulation, and Contact Safety Performance of Robotic Systems
MSMR is not aligned with ISO standards and A3 in the definition of the term “cobot.” PFL robots were called “cobots” when first introduced. This term is now obsolete, replaced by safety definitions in ISO Standard 10218 and ANSI/RIA Standard R15.06, which defines an “industrial robot” as the arm plus controller, “industrial robot system” as robot plus end effector, and “industrial robot application” as robot system, workpiece, and ancillary equipment. The distinctions between industrial
robots, systems, and applications are very important because each category has different entities responsible and accountable for safety.
Recommendation 4-1: The Measurement Science for Manufacturing Robotics Program should discontinue the use of the term “cobot” and align with International Organization for Standardization Standard 10218 and ANSI/RIA Standard R15.06.
Agility Performance of Robotic Systems
The project team could engage more with industry to ensure that the developed framework, metrics, and evaluation methods are industry-relevant. If possible, it is advisable to include more industrial participants (particularly SMEs) in the ARIAC event to ensure that it surfaces the challenges faced in industry. The results presented to the panel seemed a bit too academic. They did not seem to be very connected to the pressing industrial needs around rapid and easy re-tasking and reprogramming of robots. The tasks in ARIAC seemed to focus a lot more on the adaptability of robots rather than rapid re-tasking and reprogramming which, arguably, are more pressing needs for SMEs. The presented framework seemed to be very complex and academic. More could be done to simplify it so that it is easily digestible by industry (particularly SMEs).
Perception Performance of Robotic Systems
Vision-based human tracking systems are an industrially relevant and important technology. While mentioned briefly and with sparse detail in the read-ahead materials for the panel, it was not addressed within the scope of this project during the visit. Additionally, the project’s understanding of challenges and solutions related to 3D perception systems for robotics and needed standards is dated, based on a workshop 5 years ago co-hosted with ASTM (Committee E57). Vision-based tracking systems are also relevant to HRIs and work on mobility and exoskeleton projects.
Recommendation 4-2: The Perception Performance of Robotic Systems project should develop an intentional work statement focused on vision-based tracking systems with clear connections into adjacent projects addressing human–robot interactions and mobility and exoskeleton projects.
Now 5 years removed from the referenced co-hosted workshop, it would be helpful to have a follow-up report or another workshop, possibly both, to share progress and revisit and revise directives as appropriate.
The Perception Performance of Robotic Systems project lacks measurement methods for cognition in the context of perception. Additional emphasis on the adoption and use of open-source perception tool sets and methods—including Robot Operating System (ROS) and Point Cloud Library, as well as relevant simulation capabilities like Gazebo—would complement the project team’s skill set and body of work. It would also help connect NIST’s academic capability with low-cost solutions available for cost-effective proliferation across SMEs. It would also be an opportunity to bolster EL’s engagement with industry stakeholders (inclusive of technology providers and end users) by leveraging collaborative entities like the ROS-Industrial Consortium and the Advanced Robotics for Manufacturing Institute (ARM), because these entities and EL have a significant overlap in topics of interest.
Mobility Performance of Robotic Systems/Exosuits
Measurement science for exosuits would benefit from an expansion of the evaluation method scope to include relevant human factors and cognitive psychology implications, and such an expansion is advised.
The project’s efforts to develop and integrate sophisticated model-based controls—including AI—seem ambitious and to reach beyond what is necessary to achieve the project’s objectives.
Performance of Human–Robot Interaction
Exploring the transparency of the interactive behaviors to allow for a peer-like relationship between humans and robots would advance this project’s work. Intuitive input and output modalities can be verbal (such as common natural language) or non-verbal (such as vision and hand gestures). This may lead to greater insights into human trust, and for the robot to recognize the physical and emotional limitations of the human and adapt accordingly for a given set of manufacturing tasks. A potential challenge could be the development of measurement science to determine human—or machine—cognitive abilities.
This project would benefit from exploring emerging human-centric cognitive and physical standards such as IEEE P7017, Recommended Practice for Design-Centered Human–Robot Interaction and Governance. The dissemination of data and models such as those in the NIST repository would be important for researchers employing AI and machine learning, and for potential future research activities and collaborations. If the laboratory accepts the challenges of exploring human cognition within HRI, it would need to obtain cognitive architecture capabilities as well as associated high-performance computational hardware.
Recommendation 4-3: The Performance of Human–Robot Interaction (HRI) project should explore emerging human-centric cognitive and physical standards such as Institute of Electrical and Electronics Engineers P7017, Recommended Practice for Design-Centered Human–Robot Interaction and Governance. The data and models from this project in the National Institute of Standards and Technology repository should be disseminated to researchers employing artificial intelligence and machine learning to help drive future research activities and collaborations. If this project explores human cognition within HRI, it should obtain cognitive architecture capabilities as well as associated high-performance computational hardware.
Emergency Response Robots
This project needs to explore new opportunities to avoid being out of touch with the state of the art in this area and stakeholder needs. The publications cited by EL to the panel are pre-2017. One possibility is to explore operators’ cognitive abilities while using robotic technologies in emergency situations. Other research possibilities include advanced sensor payloads (e.g., chemical and biological sensors), computational capabilities (e.g., the use of advanced graphics processing unit architectures), and novel power techniques (e.g., lightweight batteries and “green” techniques such as solar).
The project is understaffed. If this is a NIST EL priority, additional personnel need to be hired to explore additional research opportunities. Due to the challenge of laboratory space, opportunities exist to design and develop a more agile and modular approach to obstacles, synthetic terrains, and layout representation.
Embodied Artificial Intelligence and Data Generation for Manufacturing Robots
Robotic systems that involve AI algorithms, training paradigms, and metrics are important topics to explore. These topics are largely absent from the list of publications given to the panel. Some projects lack roadmaps aligned with project objectives, showing the introduction and integration of emerging technologies, resulting in a scattershot approach with ineffective impact. Creating guidelines, standards, and other industry resources based on AI metrics will benefit from direct engagement with system integrators and end users.
Recommendation 4-4: The Embodied Artificial Intelligence (AI) and Data Generation for Manufacturing Robots project should develop and communicate clear roadmaps to achieve their stated research objectives that characterize robotic systems that involve AI algorithms, training paradigms, and metrics.
Digital Twins and Emerging Technologies for Small-and-Medium Enterprise Work Cells
During the panel meeting, the team presented a program on digital twins in manufacturing but did not have a presentation regarding this specific program aimed at robots and SME work cells. The program seems to have only a few accomplishments so far—that is, the 2023 survey of small and medium-sized manufacturers cited above. The program did not provide any information about the outcomes of the survey, even if it was preliminary. No publications, reports, or standards were cited as accomplishments from this program. It would be great to see more publications and output from the program. The program did not seem to have any team members who were familiar with the peculiar challenges of SMEs. A team member or consultant who is familiar with SMEs could be very valuable in ensuring that the program objectives and efforts are best suited to SMEs.
ASSESSMENT OF SCIENTIFIC EXPERTISE
Accomplishments
Grasping, Manipulation, and Contact Safety Performance of Robotic Systems
The EL staff in this project provides strong technical leadership and contributions to ASTM working groups, with supporting publications relevant to standards and test procedures. They are actively engaged with the research community, especially at major international conferences such as the IEEE International Conference on Robotics and Automation.
Perception Performance of Robotic Systems
Based on presentations and discussions during the tour as well as publications referenced in read-ahead materials, the scientific expertise of this team appears to be of high caliber. The team has an expert grasp of the technology domain, based on publications and presentations associated with external journals and events.
Agility Performance of Robotic Systems
The expertise of the staff has solid external recognition through the ARIAC competitions and contributions to IEEE standards working groups.
Mobility Performance of Robotic Systems/Exosuits
EL’s role in developing various aspects of the Exo Games, officially hosted by ASTM, gives them international recognition and adds credibility to their measurement science research. For both the mobile manipulator and A-UGV areas as well as wearables, the project’s strong involvement with ASTM is clear. Leadership in the development of ASTM F45 and F48 standards such as the F45.05 subcommittee on grasping and manipulation is evidence of their contributions. The specific role project personnel play and the extent of their contributions in setting standards is not clear, but this is not uncommon in collaborative standards development.
The early-career staff assigned to the mobile manipulator/A-UGV project appears well qualified and capable of working effectively toward achieving the defined project objectives. The exosuit team was enthusiastic and is successfully applying technical knowledge, even considering limitations in subject-matter expertise.
Performance of Human–Robot Interaction
This project’s staff team is on par in conference and journal publications compared to peer laboratories, such as universities in the same field. The laboratory participates in and leads various ACM and IEEE entities to generate emerging HRI standards for use in industry.
Emergency Response Robots
Before 2017, the team developed 50 test procedures, of which 25 became ASTM standards. They enhanced their expertise through the design of challenges for competitions, with strong engagement with emergency responder communities.
Embodied Artificial Intelligence and Data Generation for Manufacturing Robotics
Staff expertise is strongest in the development of sensors and the generation of data sets of objects with relevance to robotic manufacturing, especially assembly.
Digital Twins and Emerging Technologies for Small-and-Medium Enterprise Work Cells
This is a new program, so it is too early to assess scientific expertise. But the panel did not want to even imply a criticism by omitting mention of it.
Opportunities and Challenges
Grasping, Manipulation, and Contact Safety Performance of Robotic Systems
The staff could improve limited engagement with robot industry suppliers, system integrators, and end users. This would improve their understanding of improvement opportunities on the factory floor level. The staff expressed a need to better understand SME needs.
Perception Performance of Robotic Systems
The project staff’s understanding of challenges and solutions related to 3D perception systems for robotics and needed standards is dated, based on a workshop 5 years ago co-hosted with ASTM (Committee E57).
Agility Performance of Robotic Systems
The staff are too academically focused and need more industry engagement to enhance the relevance of their work.
Mobility Performance of Robotic Systems/Exosuits
This project has had strong ties with ASTM, resulting in several ASTM F45 and F48 standards. However, one staff member, recently named an ASTM Fellow specifically for his contributions, has since retired from NIST, so the future strength of the relationship with ASTM is in question.
It was difficult to determine the make-up of the full team supporting these projects and how many experienced staff are in place to lead and support technical work alongside early-career researchers from the information provided. Also, due to a recent retirement, there is ambiguity concerning the current team’s expertise because the retired researcher brought world-class expertise to the team. Of those researchers with whom the panel members interacted, the exosuit research team seems to lack an individual with a substantial background in human factors or ergonomics, and there was no indication of a plan to address this. Possibilities to address this expertise gap include collaborating with other entities or soliciting a visiting university researcher with the requisite background expertise.
Performance of Human–Robot Interaction
The project team would benefit from the inclusion of subject-matter experts in cognitive, emotional, and physiological sciences. This may lead to opportunities to explore the expansion of the workforce to include participants who are cognitively or physically challenged.
Emergency Response Robots
Contributions to this field post-2017 were not presented making it impossible to judge the current expertise of the staff.
Embodied Artificial Intelligence and Data Generation for Manufacturing Robotics
The level of technical competence is inconsistent and not always at the high standard expected from EL. In some groups, the number of contributions cited in refereed journals and conferences has been thin over the past 5 years. For example, in the publication on the EL-developed six-degree-of-freedom sensor, no quantitative data were presented on the target measurement accuracy for the sensor, how it compares with sensors commercially available to system integrators and end users, or the results achieved in laboratory testing. No information was presented on the steps required to transition the sensor to system integrators and end users interested in high-accuracy PFL robot applications. No data were presented to assess the stated objective of improving PFL robot performance in high-accuracy manufacturing applications. The absence of quantitative data is a concern, below the standards of other work in the MSMR program. The team would benefit from using the Perception Performance of Robotic Systems project as a model for technical standards. Stronger cross-team mentoring would help to raise technical competencies and contributions to the consistent high-level historically expected of the MSMR program.
Emerging AI, cognitive science, and HRI technologies will have broad impacts across all project areas. If EL is not up to speed in these areas, it risks losing relevance to industry. Expertise relevant to planning and executing the AI components of this project needs to be strengthened. Especially for projects where emerging technologies are essential components, the EL MSMR teams will need to include members with relevant academic training or research experience. Examples include reinforcement learning, generative and autonomous AI, cognitive science, and ergonomics.
Recommendation 4-5: The Measurement Science for Manufacturing Robotics Program (MSMR) should partner with universities and other research institutions with expertise in the rapidly advancing technologies relevant to its work, such as artificial intelligence, cognitive science, and human-robot interaction technologies. This would help MSMR to maintain its relevance to, and maximize its impact on, industry and U.S. competitiveness.
The MSMR program is represented externally by highly respected technical leaders who may be nearing or post-retirement. Strong contingency planning is needed, including focused recruiting, training, and retaining of staff in emerging and rapidly developing technology areas.
Digital Twins and Emerging Technologies for Small-and-Medium Enterprise Work Cells
This is a new program, and it is too early to assess it.
FACILITIES AND EQUIPMENT
Facilities
Perception Performance of Robotic Systems
This project uses a recently constructed controlled laboratory environment inside of a larger high-bay structure testing laboratory space. While somewhat confined, it appears sufficient and houses a variety of up-to-date relevant equipment including metrology equipment and PFL manipulators incorporated in the bin-picking testbed.
Adding equipment and a dedicated space in the laboratory for human tracking systems capabilities is suggested. As referenced in the section “Assessment of Technical Programs,” this is an important industrial relevant technology, and this capability would be especially valuable to the Mobility Performance of Robotic Systems project, based on its physical and topical proximities. Additional collaboration with other project laboratories to share robot manipulator resources if and as needed would also be beneficial.
Mobility Performance of Robotic Systems/Exosuits
Both the A-UGV/mobile manipulator and exosuit teams appear to be successfully accomplishing major portions of the project’s stated objectives, despite facility limitations. The new A-UGV high-fidelity optical tracking system for reliable ground truth is an indicator of an adequate non-labor budget. A variety of active and passive exosuit measurement devices have also been obtained and used in test method development.
Exosuit facilities appear make-shift, being in the back corner of a high bay used for testing structures with inconsistent environmental control and do not lead to a perception of world-class work.
The mobile manipulator facilities, while apparently adequate for currently defined work, appear make-shift and do not give world-class perception. One aspect lacking is the ability to emulate the potential variety of environmental conditions related to mobile manipulation use cases, especially in terms of lighting and other visibility conditions.
Equipment
Performance of Human–Robot Interaction
The project’s facility contains state-of-the-art equipment and computational hardware that is on par with other research facilities and is adequate to achieve the stated research objectives. Besides robots, sensors, and commercially available augmented reality and virtual reality devices, the laboratory has a motion capture and rigid body tracking system to enable the initial exploration of collaborative interactions beyond existing techniques based on force and torque. All equipment as well as the facility are relevant and up to date.
EFFECTIVENESS OF DISSEMINATION EFFORTS
Accomplishments
Grasping, Manipulation, and Contact Safety Performance of Robotic Systems
There are two primary dissemination mechanisms. The first mechanism is active contributions to ASTM Working Groups in Committee F45 on Robotics, Automation, and Autonomous Systems, eventually leading to industry standards. Team members also present new standards and supporting
technologies at major industry events such as A3 Automate. The second mechanism is the development of ATBs that are used by research teams worldwide and leading conferences.
Perception Performance of Robotic Systems
Dissemination of this group’s work is through participation in industry events such as A3 Automate and The Vision Show and co-hosting ASTM Committee E57: 3D Imaging Systems workshops and meetings. Dissemination is also accomplished through the development of the technical basis for new standards in ASTM Committee E57. For example, a new standard for point cloud data formatting has been published.
Agility Performance of Robotic Systems
Dissemination mechanisms include contributions to ARIAC and engagement with IEEE standards working groups. ARIAC participants are mainly students and post-doctoral researchers.
Mobility Performance of Robotic Systems/Exosuits
The International Exo Games and participation in ASTM Committee F48 on Exoskeletons and Exosuits are the primary means of dissemination and validation of the 10 defined testing methods thus far. Project results and test methods for mobile robots and mobile manipulators are disseminated through new standards in ASTM Committee F45 on Robotics, Automation, and Autonomous Systems.
Embodied Artificial Intelligence and Data Generation for Manufacturing Robotics
The principal dissemination mechanism is through the NIST MOAD data set. One component of this project uses the NIST ATBs developed by the Grasping, Manipulation, and Contact Safety Performance team.
Digital Twins and Emerging Technologies for Small-and-Medium Enterprise Work Cells
The MOAD data sets provide an effective mechanism for dissemination to support advanced robotic manufacturing research and product development, as part of the larger ATB initiative. The NIST MOAD ATB Dataset v1 was released in January 2024. The ATBCOMP data set was released to support the 2024 IEEE International Conference on Robotics and Automation competition.
Emergency Response Robots
Participation in ASTM Committee E54.09 on Response Robots is the primary means of dissemination of test methods of response robots.
Opportunities and Challenges
Grasping, Manipulation, and Contact Safety Performance of Robotic Systems
The dissemination of standards information is largely limited to ASTM working groups and presentations at annual events such as the annual A3 Automate Show. Beyond these venues, there is little awareness of this MSMR work, especially among robot system integrators and subject-matter experts. Furthermore, the use of ATBs is largely limited to the research community, with limited evidence of industry impact or use. More broadly, MSMR staff expressed a need for a better understanding of real-world requirements and technology gaps on the factory floor. Expanded engagement with manufacturing innovation institutes (MIIs)—particularly the ARM Institute (Advanced Robotics for Manufacturing),2
___________________
2 For more information, see the ARM Institute website at https://arminstitute.org, accessed December 11, 2024.
MxD (Manufacturing x Digital),3 and CESMII (the Smart Manufacturing Institute)4—would aid in the dissemination of research results and also garner feedback about what is needed by stakeholders. This could also foster a shared awareness of technical roadmaps between EL and MIIs.
Agility Performance of Robotic Systems
No information was presented about any direct engagement with industry robot suppliers, integrators, or end users. If there is not any, this is not only a significant lack of dissemination, but potentially a blind spot for this project.
Mobility Performance of Robotic Systems/Exosuits
The exosuits team has collectively published seven external journal papers, articles, or conference papers during the assessment period and has issued seven papers to internal NIST publications, although the internal papers’ impact is unclear. More engagement with and presentations at industry-focused events such as A3 Automate and ARM is advisable. Publication of their work on NIST’s domain website is passive; additional dissemination paths such as social media marketing via LinkedIn, lunch-and-learn style webinars, and the like could be valuable dissemination modes.
Performance of Human–Robot Interaction
More engagement with MEPs through more frequent or substantial leveraging of the MEPAssisted Technology and Technical Resource program could help with dissemination to SMEs, broadening and deepening impact. The focus areas of this project are highly relevant to manufacturers looking to implement automation to help them with high-mix, low-volume applications.
Emergency Response Robots
No information was presented on dissemination post-2017. This, itself, is a weakness.
Embodied AI and Data Generation for Manufacturing Robotics
No direct engagement with industry robot suppliers, integrators, or end users was presented.
Recommendation 4-6: All projects in the Measurement Science for Manufacturing Robotics Program should expand their engagement with manufacturing innovation institutes to bolster dissemination and engagement with large and small system integrators and end users.
REFERENCES
DeLoitte. 2024. “US Manufacturing Could Need as Many as 3.8 Million New Employees by 2033, According to Deloitte and The Manufacturing Institute.” April 3. https://www2.deloitte.com/us/en/pages/about-deloitte/articles/press-releases/us-manufacturing-could-need-new-employees-by-2033.html.
NAM (National Association of Manufacturers). 2024. “Manufacturing Job Openings Decline.” May 1. https://nam.org/manufacturing-job-openings-decline-30910.
___________________
3 For more information, see the MxD website at https://www.mxdusa.org, accessed December 11, 2024.
4 For more information, see the CESMII website at https://www.cesmii.org, accessed December 11, 2024.
