Summary

Decades of research and practice in the safety¹ engineering of physical systems has produced a body of knowledge that includes rigorous approaches, tools, and artifacts for modeling analyzing and measuring system performance which is intended to ensure a societally acceptable level of safety. The emergence of the field of cyber-physical systems further extended this work to improve the safety of cyber-enabled control of physical systems—with impressive results. These safety-critical systems are some of the most dependable and reliable engineering systems that have ever been built. They are engineering marvels that enhance and transform human capabilities.

Recent accelerating progress in artificial intelligence (AI), specifically machine learning (ML),² is enabling new capabilities, including prediction, decision making, perception, language processing, interactivity, and collaboration with humans. These capabilities are transforming many sectors of society, including health care, energy, transportation, manufacturing, education, and many more. However, while ML is being used today in groundbreaking engineering efforts to enhance and augment system capabilities and interactions with humans and the physical world, its implications—both positive and negative—are both highly significant and not yet well understood. Extending safety systems engineering to cyber-physical systems that include ML-enabled components brings a new set of challenges that suggest changing how this engineering is conceived,

___________________

¹ For the purposes of this study, safety refers to the preservation and protection of physical well-being of persons, property, and environment. Relatedly, safety-critical systems are defined as engineered systems whose failure or malfunction can result in physical harm to people, the environment, or property. A context note on AI safety is provided in Chapter 1.2.6.

² This study considers only machine learning (ML) and its application in safety-critical systems. The broader term artificial intelligence (AI), which encompasses ML as well as other AI techniques, is sometimes used in the report only to be consistent with its use in popular literature; for example, when discussing AI policies and regulations or embodied AI.

approached, and carried out. These challenges are especially urgent owing to the continued rapid development of ML technologies and their widespread deployment in commercial applications. For example, today ML is a key enabler of automated vehicles even as some risk factors are yet to be fully understood or successfully mitigated. Similarly, the digital transformation of smart manufacturing, where cyber-physical systems are increasingly becoming more autonomous and interconnected, raises new challenges for both safety and engineering process management.³

To help advance understanding of safety properties in safety-critical systems that use ML (see Box S-1), the National Academies of Sciences, Engineering, and Medicine convened a consensus study committee to consider key issues regarding safety adaptation, metrics, testing and evaluation, and new approaches to risk assessment and mitigation (see Appendix A). The Committee on Using Machine Learning in Safety-Critical Applications: Setting a Research Agenda focused its work on safety-critical cyber-physical systems such as automated vehicles, robots in health care, and the power grid. The committee did not focus on the use of ML in purely cyber systems (e.g., chatbots) but acknowledges that failure or misuse of such systems can also result in harm. This work is of necessity a snapshot of knowledge and practices as of the end of 2024. Owing to the rapid pace of development in ML technologies, new ML approaches as well as additional safety challenges and opportunities are likely to emerge after the report is published.

___________________

³ A. Forcina and D. Falcone, 2021, “The Role of Industry 4.0 Enabling Technologies for Safety Management: A Systematic Literature Review,” Procedia Computer Science 180:436–445.

The research and development communities for safety-critical systems and ML differ in community norms and standards, governance approaches, and culture. For example, research in image recognition has focused primarily on the performance of individual components rather than combined systems. Today’s best-performing computer vision algorithm achieves an accuracy of 97.0 percent correct classifications on the ImageNet object recognition benchmark.⁴ A common baseline accuracy for safety-critical system that incorporates an algorithmic component is that the system must have an error rate of 10⁻⁹. The disparity between an ML algorithm’s 97.0 percent accuracy and the safety system expectation of 99.9999999 percent cannot be bridged by existing ML methods alone. The challenge facing safety engineers is thus to design systems that can achieve the desired failure rate or risk reduction factor even though individual ML-based system components cannot. For example, systems engineers might add fault-detection mechanisms or additional safeguards, such as radar and lidar sensors that can perform well when camera-based object recognition fails. Alternatively, safety engineers could consider whether an ML-based object recognition component can reliably quantify its own uncertainty on a case-by-case basis. If the reported uncertainty falls below a predetermined safety threshold, the component can guarantee a sufficiently lower error. In other words, integrating ML components into safety-critical applications requires safety engineers to understand the failure modes of ML systems and develop new methods to detect and mitigate those failures. This shift demands changes in research, training, and engineering practice.

The “safety gap” is significant and difficult to bridge due to differences of culture and approach. ML research does not generally follow traditional system engineering methodologies, such as system modeling, development of formal system specifications, and failure mode and effect analysis. Conversely, safety engineering processes and safety standards demand that system components be interpretable, traceable, highly accurate, and robust—factors that are all key areas of active research in ML.

Efforts are under way to bridge this gap. Researchers and industry practitioners are already working on new ways to evolve safety culture toward integrating ML components into safety-critical systems. In the automotive industry, for example, technical, regulatory, and best practice advances are enhancing capabilities (e.g., anomaly detection) and actively restricting features (e.g., turning off autopilot) to reduce anticipated hazards in higher-risk scenarios.

ML algorithms, however advanced, are only components of an overall system, and integrating an ML-based component into a larger system raises many engineering challenges. The ML research community has focused almost entirely on the performance

___________________

⁴ M. Tan and Q.V. Le, 2019, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks,” Proceedings of Machine Learning Research 97:6105–6114.

of ML components in isolation. Therefore, a key paradigm shift in research is needed to make ML “fit for safety” by addressing additional capabilities required to integrate ML components into safety-critical systems. This includes research to improve the following qualities of the ML component:

• Uncertainty quantification: Ability to quantify predictive uncertainty for individual inputs such that if uncertainty is low, high levels of accuracy can be guaranteed; and if uncertainty is high, mitigations must be engaged.
• Traceability and explanation: Ability to trace the cause(s) of a system output error back to either the relevant training data or to violations of the operational domain.
• Competence modeling: Ability to detect when an input lies outside the domain of competence of the trained subsystem. For example, the perceptual system of a medical diagnosis system should detect when the X-ray image exhibits some novel phenomenon (e.g., a new disease) that the system has not been trained to recognize.
• Robustness: Ability to create learned components that can perform accurately in the presence of a known range of perturbations or disturbances in the input.

Foundation models (FMs), a recent development in ML, exhibit a versatility that could be advantageous in safety-critical settings, especially in applications where challenges and contexts can vary. These are models with billions of parameters trained on immense data sets that can be fine-tuned on task-specific data to achieve high performance in narrow applications. FM’s breadth of knowledge can also provide valuable support for human–machine interactions. Large language models (LLMs), a type of FM, have also been shown to be effective at translating natural language specifications to formal machine specifications, potentially significantly advancing human–machine interaction and teaming. On the other hand, FMs are not without their own major shortcomings. The sheer scale of FMs’ training data, and of the model itself, have made it thus far virtually impossible to support explainability and traceability—key attributes in safety system development. Researchers have also shown that LLMs can exhibit highly problematic behaviors from a safety perspective such as hallucination (generating false results) and sycophancy (results influenced by the user’s perceived desires)—and that they can be intentionally manipulated. Therefore, while FMs hold immense potential, at present their performance falls short of safety standards and should be carefully considered when used in safety-critical systems.

While research may improve the safety of ML-enabled systems, ultimately, the decision to use a new technology is up to its users. It is important for consumer organizations, public interest groups, system integrators, and testing laboratories to have the right information available to them to help inform users of the risks. Toward this end, disclosing ML algorithms and models, training, testing and evaluation data and methods, and continual performance monitoring are examples of steps toward transparency. Concurrently, regulatory bodies and public-interest entities should also prioritize similar levels of transparency and continue to monitor and disclose ML technology’s performance data, solicit public input, and implement a transparent process for updating safety standards and regulations for ML-enabled systems in relevant application domains.

FINDINGS

The findings that follow are intended to help guide those formulating policies, advancing industry practices, or setting research priorities—all of which will contribute to enhance the performance of safety-critical systems with ML-enabled components.

On the Emergence of Machine Learning in Safety-Critical Systems

Finding S-1: Integration of ML in safety-critical cyber-physical systems exposes a fundamental challenge: The ML and safety engineering communities have different methods, standards, and cultures that must be reconciled if the performance and safety potential of ML is to be realized.

On the State of the Art and Promise of Machine Learning

Finding S-2: Although ML now matches or exceeds human performance in many tasks, its deployment in physical systems presents both opportunities and risks. The opportunities include novel applications, enhanced efficiency, and improved human–machine collaboration, while the risks include classification errors and vulnerability to malicious attacks, among others.

The advent of large-scale data sets, scalable neural architectures, and increased computational power has revolutionized deep learning and unlocked new capabilities that surpass those of previous approaches in ML and other relevant fields. The potential impact of ML in the physical world is profound, including bolstering efficiency, enhancing performance, personalizing experiences, and fostering greater collaboration between humans and machines. These new capabilities are being applied to enhance

cyber-physical systems across virtually all sectors. The synergy of ML with intelligent infrastructure promises a future where infrastructure is not just interconnected but also possesses the ability to learn and improve performance. If done correctly, this has the potential to advance the capabilities and resilience of the U.S. intelligent infrastructure. In health care and medicine, ML, including recent advancements in FMs, is enabling new applications ranging from robot-assisted medical procedures to language-based interaction with ML chat systems. These applications have the potential to improve patient outcomes, reduce costs, and lower barriers to expert care. In robotics, safe deployment of general-purpose robotic systems with human-like dexterity and adaptability could unlock immense value across various manufacturing sectors by effectively collaborating with human workers to perform tasks, reducing hazardous exposure for workers and helping to address shortages in skilled human labor. In the automotive industry, ML is already today an indispensable technology. It is key to enabling future systems to continue moving to higher levels of automation.

While application of ML has resulted in observable benefits to society, there are numerous safety risks that emerge when ML is integrated into real safety systems. These risks occur across many different domains and can result in loss of life and damage to property or the physical environment. For example, a perception mistake could lead to automotive accidents resulting in death, and misinterpretation of medical data could result in mistreatment and worsen health outcomes.

On the Challenges of Engineering Machine Learning for Safety-Critical Systems

Finding S-3: Current ML components fall short of safety-critical standards because they rely on statistical assumptions that discount rare events and cannot guarantee consistent performance across all operating conditions. To close this gap, ML components must be redesigned to maintain verified levels of accuracy throughout their entire operational domain.

System components constructed with ML-enabled capabilities exhibit significant error rates both because of incomplete sensing of the environment (e.g., occluded objects in computer vision) and because of their inherently statistical nature. Ultimately, ML components will never be perfect, and system design must take this into consideration. Systems may also rely on the outputs of ML components for decision making. Since ML models inherently contain uncertainty, the system thus needs to take actions to hedge against uncertainty or to reduce uncertainty. In many cases, ML-enabled safety-critical systems operate in an open world where novelty will be encountered. These systems need to implement an “outer loop” in which novelties are detected and characterized

and the system is extended, through data collection and retraining, to properly handle the discovered novelties.

Unlike in traditional system design where each component has a written functional specification, the specification for each learning component is a set of training examples. Hence, ML components need to be specified in terms of the amount of training data and the dimensions of variability that the data must exhibit. New data engineering tools are needed to support the specification process. Furthermore, ML components need to be tested to ensure that they have not learned spurious correlations. Advances in test data design and in ML explanation capabilities are needed to achieve this goal.

There will be times when the system is operating outside of its operational domain (OD) (the domain covered by training data). In these regions, the system will be operating under uncertain conditions. To support decision making under this uncertainty,⁵ ML components need to provide calibrated representations of their uncertainty. Additionally, the validity of ML components rests on assumptions, particularly for novelty detection and operational domain detection, which cannot be checked prior to deployment. Specifically, validity assumes that the training data exercises all important directions of variation along which novel and out-of-operational domain data will arise.

On a New Research Agenda

Finding S-4: Safely integrating ML components into physical systems requires a new engineering discipline that draws from ML and systems engineering, and a new research agenda in which ML researchers adopt a broader systems perspective and safety engineers embrace the distinctive attributes of ML.

The new research agenda for integrating ML into safety-critical systems, by necessity, draw from several fields of study, including data science and engineering, algorithms, and testing and evaluation. Key research gaps include the following:

• New and improved tools are needed for specifying data requirements for ML components and for collecting data to meet those requirements.
• Multiple benchmark data sets and learning tasks are needed to advance research on ML-enabled safety-critical systems.
• New learning algorithms and associated theory are needed for scenario-based safety-critical machine learning.

___________________

⁵ There are two uncertainties referenced in this section. The previous paragraph discusses uncertainty due to inaccuracies of the ML model—that is, inherent to the model—whereas, this paragraph discusses uncertainty due to the system operating in an unrecognized environment.

• Research is needed to develop practical methods of preventing—or at least detecting—the learning of non-causal relationships.
• Scalable methods are needed for explanation and attribution of machine learning predictions.
• A general methodology is needed for representing uncertainty over complex predictions.
• A general methodology is needed for calibrating complex uncertainty representations.
• Although there has been progress in verifying desired properties for neural networks or learning enabled systems, critical research is still needed to improve the scaling of these methods and to explore additional approaches beyond scaling.
• Research is needed to develop application-centric evaluation methodologies for reliability, robustness, and resilience across the system life cycle.
• Research is needed to develop metrics and guardrails for specific performance variables, operating scenarios, and tasks. Generalized performance metrics are not generally sufficient for evaluating safety.

On Societal Considerations Toward Building Trust

Finding S-5: Although guidance for governing ML in safety-critical systems has progressed, new standards, regulations, and testing methods are needed to address both cross-cutting and domain-specific safety challenges.

Governments often establish regulations where protections are needed to ensure public trust, security, and health and safety. There has been meaningful progress in developing guidance for ML. However, these measures alone cannot adequately or independently address the stringent requirements of safety-critical systems. Furthermore, the anticipated applications of ML cross into many regulated safety-critical domains and continue to drive new and broad policy and regulatory debate. The situation can be more complex when new horizontal (cross-cutting) ML regulations intersect with long-established regulatory and compliance methods in specific vertical domains. Coordinating and harmonizing horizontal ML standards with vertical safety-critical system standards is essential for integrating ML components into safety systems effectively. Consensus standards efforts such as ISO/IEC JTC1 SC42, NIST’s Artificial Intelligence Safety Institute, and the regional standards work in CEN/Cenelec JTC21 are important first steps toward engineering safety-critical system frameworks with ML components.

Implementing processes for learning from in-field incidents (including near misses) is essential for building and maintaining trust in ML-based safety-critical systems. These

processes should include analysis of causes for such incidents by an independent body and enforcing post-delivery upgrades of deployed systems with a focus on eliminating the known causes for such incidents. Furthermore, proficient testing capabilities are needed to ensure the safe deployment of ML technologies in safety-critical applications. This may involve, for example, the development of representative testing grounds and highly faithful digital twins of such systems and their environment.

And finally, to educate the next generation of researchers and engineers on how to build ML-enabled safety-critical systems, graduate-level courses and curricula are needed that emphasize a holistic systems perspective, building on and integrating competencies on ML design, IT design, systems safety and security, and human–machine cooperation, among others.

* * *

The integration of machine learning into safety-critical physical systems represents an extraordinary opportunity for automation but also a significant challenge for safety engineering. Successfully bridging the divide between ML and safety engineering communities requires new scientific approaches, standards, and educational initiatives. These efforts will be essential to realize ML’s potential to enhance safety and performance across transportation, health care, manufacturing, and other critical domains while maintaining the stringent safety standards that society demands.

This page intentionally left blank.