WASHINGTON — A new report from the National Academies of Sciences, Engineering, and Medicine examines how the U.S. Department of Energy could use foundation models for scientific research, and finds fusing these models with traditional computational methods could bring a paradigm shift to scientific discovery. The report makes recommendations for how the agency can pursue increased use of such models in research.

Foundation models are large-scale AI neural networks that are trained on vast amounts of data — often trillions of individual data points — and after fine-tuning, they are capable of learning new ways of modeling information and performing a range of tasks. Foundation models are a departure from many AI tools used in scientific research, which are often designed for specific purposes.

In scientific discovery, foundation models not only have the advantage of being able to handle huge volumes of heterogeneous data, but they also have characteristics like the ability to work across different kinds of tasks, self-supervise their own training, and have architectures that can serve multiple purposes. These characteristics potentially enable foundation models to generate findings and discern patterns within datasets at a volume that exceeds — by orders of magnitude — the computing and storage capacity of more traditional computational methods used by DOE, and even of previous machine learning models. However, much work is needed to strengthen assurance, verification, and validation in foundation models, and to quantify uncertainty in their output.

Traditional computational methods form the bedrock of trusted model simulations that enable predictive science across some of the most complex systems that DOE studies, such as materials physics and Earth’s systems. These models are grounded in physical laws and are both verified and validated, characteristics that are indispensable for safety-critical research, such as research on nuclear systems.

Rather than seeing foundation models as a replacement for traditional models, the report urges a synergistic integration of the two — which may bridge the gap between predictive modeling and interpretive reasoning, bringing researchers closer to having models that not only solve complex problems but also explain them.

“Foundation models hold great promise for scientific discovery, even as their nascent stage of development means they carry limitations and challenges,” said Dona Crawford, retired associate director for computation, Lawrence Livermore National Laboratory, and chair of the committee that wrote the report. “Our report offers a path forward for how the Department of Energy could go about both developing and using this potentially transformative technology at the cutting edge of scientific discovery.”

Foundation Models at DOE

The report says DOE should continue to invest in developing its foundation models, particularly in areas of strategic importance and where the agency already has an advantage. For example, DOE has robust physical scientific data on which to train and test foundation models; the agency can attract top talent; and it has the opportunity to leverage autonomous AI systems to help run scientific labs. DOE should prioritize hybridizing foundation models, but not abandon its expertise in computational methods — continuing investment in software and infrastructure.

Among other recommendations, the report says DOE should establish and enforce standardized protocols and develop benchmarks for training, documenting, and reproducing foundation models for science, and pursue partnerships with industry and academia to address national mission goals.

The study — undertaken by the Committee on Foundation Models for Scientific Discovery and Innovation — was sponsored by the U.S. Department of Energy. The National Academies of Sciences, Engineering, and Medicine are private, nonprofit institutions that provide independent, objective analysis and advice to the nation to solve complex problems and inform public policy decisions related to science, engineering, and medicine. They operate under an 1863 congressional charter to the National Academy of Sciences, signed by President Lincoln.

Contact:
Megan Lowry, Media Relations Manager
Office of News and Public Information
202-334-2138; email news@nas.edu