Recently completed
A committee-supported project or activity that has been completed and for which output dissemination has begun. Its committee has been disbanded and closeout procedures are underway.
Topics
Machine learning components are enabling advances in self-driving cars, the power grid, and robotic medicine, but what are the implications for safety?
Decades of research and practice in safety engineering have created rigorous approaches to design, model, and analyze systems that meet stringent safety criteria, but extending such techniques to include machine learning components brings new challenges.
This National Academies’ report explores ways to design machine learning algorithms that align with safety engineering standards, noting that this will require changes in research, training, and engineering practice—and a shift away from focusing on the performance of machine learning algorithms in isolation.
Featured publication
Consensus
·2025
Advances in artificial intelligence, and specifically in machine learning, are enabling new capabilities across nearly every sector of the economy. Many of these applications - such as automated vehicles, the power grid, or surgical robots - are safety critical: where malfunctions can result in harm...
View details
Description
A National Academies of Sciences, Engineering, and Medicine study will explore the trustworthiness of machine learning (ML), especially very large or complex models, in safety-critical applications. The study will consider such questions as:
- What are core principles of trustworthiness in safety-critical systems? What adaptations are needed to accommodate ML models?
- What metrics of trustworthiness are currently used to assess safety-critical systems that do not rely on ML? Which are applicable to systems that rely on ML and how can they best be applied? What new metrics are needed for systems that rely on ML? For example, what does it mean for a system that relies on ML to be correct?
- How should systems relying on one or more ML models be tested and evaluated? What types of assurances are possible? How can reliability requirements be satisfied when a system employs nontransparent models? What impact do limits on the explainability of ML models have on evaluating and ensuring safety?
- How does the robustness of today’s ML models compare with the level of robustness expected in non-ML systems that are certified or otherwise approved for safety-critical applications? What are opportunities to better understand and enhance the robustness of ML models?
- How do traditional approaches for achieving trustworthiness such as testing need to be modified for safety-critical systems that rely on ML? What new formalisms are needed to describe and assess ML trustworthiness?
- How can monitoring and run-time verification be used with systems that rely on ML to identify potentially unsafe conditions and enable a system or its human operators to take fail-safe action?
- What investments in research on ML in safety-critical systems are needed to complement ongoing work on related topics in the ML research community?
- How can the cost of ML failures in safety-critical systems be quantified and measured?
The committee’s report will describe the present state-of-the-art in approaches to engineering safety-critical systems (both involving ML and not) and identify research that would (1) enhance understanding of the challenges in building safe systems that rely on ML and (2) foster improvements to the safety of systems that rely on ML to be improved. It may provide findings and conclusions as appropriate but will not provide recommendations.
Collaborators
Sponsors
Open Philanthropy Project
Staff
Jon Eisenberg
Lead
Tho Nguyen
Lead
Shenae Bradley
