Assessing the Reliability of Complex Models

Consensus

Download PDF

Paperback - $37.80
Ebook - $30.59

Advances in computing hardware and algorithms have dramatically improved the ability to simulate complex processes computationally. Today's simulation capabilities offer the prospect of addressing questions that in the past could be addressed only by resource-intensive experimentation, if at all. Assessing the Reliability of Complex Models recognizes the ubiquity of uncertainty in computational estimates of reality and the necessity for its quantification.

As computational science and engineering have matured, the process of quantifying or bounding uncertainties in a computational estimate of a physical quality of interest has evolved into a small set of interdependent tasks: verification, validation, and uncertainty of quantification (VVUQ). In recognition of the increasing importance of computational simulation and the increasing need to assess uncertainties in computational results, the National Research Council was asked to study the mathematical foundations of VVUQ and to recommend steps that will ultimately lead to improved processes.

Assessing the Reliability of Complex Models discusses changes in education of professionals and dissemination of information that should enhance the ability of future VVUQ practitioners to improve and properly apply VVUQ methodologies to difficult problems, enhance the ability of VVUQ customers to understand VVUQ results and use them to make informed decisions, and enhance the ability of all VVUQ stakeholders to communicate with each other. This report is an essential resource for all decision and policy makers in the field, students, stakeholders, UQ experts, and VVUQ educators and practitioners.

Publication information

144 pages · 8.5 x 11 · paperback
ISBN Paperback: 0-309-25634-8
ISBN Ebook: 0-309-25635-6
DOI: https://doi.org/10.17226/13395

Suggested citation

National Research Council. 2012. Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification. Washington, DC: The National Academies Press.

Import this citation to:

Marvin L. Adams

Marvin L. Adams is the HTRI Professor of Nuclear Engineering and the Director of the Institute for National Security Education and Research at Texas A&M University. His research has focused on many aspects of computational science and engineering, including discretization methods, iterative methods, parallel algorithms, and quantifying predictive capability. He has served as a consultant to Lawrence Livermore National Laboratory, Sandia National Laboratories, and Los Alamos National Laboratory, and has served on a variety of review and advisory committees and panels for the laboratories, DOE, and other government organizations. Dr. Adams earned his B.S from Mississippi State University in 1981 followed by M.S. (1984) and Ph.D. (1986) degrees from the University of Michigan, all in nuclear engineering. From 1977 to 1982 he worked at Tennessee Valley Authority’s Sequoyah Nuclear Plant and its support office. He joined Lawrence Livermore National Laboratory after completing his graduate work in 1986. He left Livermore in 1992 for the faculty position that he continues to hold at Texas A&M University. In 2006-2007 he founded and directed the Center for Large-scale Scientific Simulation at Texas A&M, and from 2005 until 2009 he served as Associate Vice President for Research.

David M. Higdon

David M. Higdon is Group Leader for the Statistical Sciences Group at Los Alamos National Laboratory. He is nominated as the second co-chair of the committee. He is an internationally recognized expert in Bayesian statistical modeling of environmental and physical systems. He has also led numerous programmatic efforts at LANL in the quantification of margins and uncertainties and uncertainty quantification. His recent research has focused on simulation-aided inference in which physical observations are combined with computer simulation models for prediction and inference.
His research interests include space-time modeling; inverse problems in physics, hydrology and tomography; inference based on combining deterministic and stochastic models; multiscale models; parallel processing in posterior exploration; statistical modeling in physical, environmental and biological sciences; Monte Carlo and simulation based methods.

James O. Berger

James O. Berger (NAS) is Arts and Sciences Professor of Statistics and Duke University. He was a faculty member in the Department of Statistics at Purdue University until 1997, at which time he moved to the Institute of Statistics and Decision Sciences (now the Department of Statistical Science) at Duke University. He has also been Director of the national Statistical and Applied Mathematical Sciences Institute since 2002. He was president of the Institute of Mathematical Statistics from 1995-1996, chair of the Section on Bayesian Statistical Science of the American Statistical Association in 1995, and president of the International Society for Bayesian Analysis during 2004. He has been involved with numerous editorial activities, including co-editorship of the Annals of Statistics during the period 1998-2000, and has organized or participated in the organization of over 35 conferences. Among the awards and honors Berger has received are Guggenheim and Sloan Fellowships, the COPSS President’s Award in 1985, the Sigma Xi Research Award at Purdue University for contribution of the year to science in 1993, the Fisher Lectureship in 2001, election as foreign member of the Spanish Real Academia de Ciencias in 2002, election to the USA National Academy of Sciences in 2003, award of an honorary Doctor of Science degree from Purdue University in 2004, and the Wald Lectureship in 2007. Berger’s research has primarily been in Bayesian statistics, foundations of statistics, statistical decision theory, simulation, model selection, and various interdisciplinary areas of science and industry, especially astronomy and the interface between computer modeling and statistics.

Derek Bingham

Derek Bingham is an Associate Professor and the Canada Research Chair in Industrial Statistics in the Department of Statistics and Actuarial Science at Simon Fraser University. He received his PhD from the Department of Mathematics and Statistics at Simon Fraser University in 1999, winning the Governor General’s Gold. After graduation he joined the Department of Statistics at the University of Michigan as an Assistant Professor. He moved back to Simon Fraser in 2003. In addition, since 2002 he has held a Faculty Affiliate position at the Los Alamos National Laboratory. The main focus of Derek Bingham’s research is the development of statistical methodology for the design and analysis of industrial experiments. This work focuses on developing new methodology for: (1) design and analysis of computer experiments; (2) Bayesian design and analysis of experiments in industrial problems such as optimal screening designs, response surface optimization and optimal robust parameter designs for product variation reduction; and (3) design and analysis of fractional factorial plans for multi-stage processes. Most of the problems that motivate this research are the result of scientific collaborations. For example, recent work relating to experiments on computer simulators is the direct result of interaction with scientists at Los Alamos National Laboratory and the Center for Radiative Shock Physics at the University of Michigan. Prof. Bingham also is the Principle Investigator on the NPCDS funded project on the Design and Analysis of Computer Experiments for Complex Systems as well as NSERC Discovery and Accelerator grants on the design of industrial experiments. To facilitate interaction with industry, he created the Industrial Statistics Laboratory within his Department. The Lab provides an environment for faculty, research scientists, students, post-docs and visitors to collaborate on research problems.

Wei Chen

Wei Chen is the Wilson-Cook Chair and Professor in Engineering Design at Northwestern University. She is affiliated with the Segal Design Institute as a Faculty Fellow and is a Professor in the Department of Mechanical Engineering, with courtesy appointment in the Department of Industrial Engineering and Management. As a director of the Integrated Design Automation Laboratory, her current research involves issues such as simulation-based design under uncertainty, model validation, stochastic multiscale analysis and design, robust shape and topology optimization, multidisciplinary optimization, consumer choice modeling and enterprise-driven decision-based design. She is the co-founder and Director of the interdisciplinary doctoral cluster in Predictive Science and Engineering Design at Northwestern, a program aiming for integrating scientific, physics-based modeling and simulation into design of innovative “engineered” systems. She is the recipient of the 1996 US National Science Foundation Faculty Early Career Award and the 1998 American Society of Mechanical Engineers (ASME) Pi Tau Sigma Gold Medal achievement award. She is also the recipient of the 2005 Intelligent Optimal Design Prize and the 2006 SAE Ralph R. Teetor Educational award. Dr. Chen is a Fellow of ASME, Associate Fellow of AIAA, and a member of SAE. She is an elected member of the ASME Design Engineering Division Executive Committee and an elected Advisory Board member of the Design Society, an international design research community. She serves as the review editor of Structural and Multidisciplinary Optimization and an Associate Editor of the Journal of Engineering Optimization. In the past, she served as the Chair and member of the ASME Design Automation Executive Committee during 2002-2007 and was an Associate Editor of the ASME Journal of Mechanical Design (03-06).

Roger Ghanem

Roger Ghanem is professor in the Viterbi School of Engineering at the University of Southern California. Dr. Ghanem has a PhD in Civil Engineering from Rice University and had served on the faculty of the Schools of Engineering at SUNY-Buffalo and Johns Hopkins before joining USC in 2005. Ghanem’s research is mainly in the area of computational science and engineering with a focus on uncertainty quantification and prediction validation in complex systems. His recent interests include sustainability of coupled interacting systems such as SmartGrid and the interface of human and natural environments, as well as the predictability of physical behaviors exhibiting coupling between multiple underlying phenomena and scales. Ghanem has over 100 refereed journals publications in the general areas of stochastic modeling and computations and dynamical systems. He has received several awards for his teaching and research, is the founding editor of Lecture Notes in Mechanics (ASCE-EMI), and serves on the advisory board for a number of professional Journals. He currently serves on the Board of Governors of the Engineering Mechanics Institute of the American Society of Civil Engineers (ASCE-EMI).

Omar Ghattas

Omar Ghattas is the John A. and Katherine G. Jackson Chair in Computational Geosciences and Professor of Geological Sciences and Mechanical Engineering at the University of Texas, Austin. He is also Research Professor in the Institute for Geophysics; Director of the Center for Computational Geosciences in the Institute for Computational Engineering and Sciences; Professor of Biomedical Engineering and Computer Sciences (by courtesy); Co-Chief Applications Scientist for the 580 Teraflops NSF Track 2 supercomputer at the Texas Advanced Computing Center; and Director of the KAUST-UT Austin Academic Excellence Alliance. From 1989 to 2005, he was a professor at Carnegie Mellon University. He has been a visiting professor at the Institute for Computer Applications in Science and Engineering (ICASE) at NASA-Langley Research Center, the Center for Applied Scientific Computing (CASC) at Lawrence Livermore National Laboratory, and the Computer Science Research Institute (CSRI) at Sandia National Laboratories. His research interests are in forward and inverse modeling, and optimal design and control, of complex systems in the geological, mechanical, and biomedical engineering sciences, with particular emphasis on large-scale simulation on parallel supercomputers. He received the 1998 Allen Newell Medal for Research Excellence, the Supercomputing 2002 Best Technical Paper award, the 2003 Gordon Bell Prize for Special Accomplishment in Supercomputing, the 2004/2005 CMU College of Engineering Outstanding Research Prize, the SC2006 HPC Analytics Challenge Award, and the TeraGrid 2008 Capability Computing Challenge Award, and was a finalist for the 2008 Gordon Bell Prize. His recent professional activities have included organization of 10 conferences and workshops in computational science and engineering; delivering 15 keynote or plenary talks at major international conferences; Program Director for SIAM’s Computational Science & Engineering Activity Group; founding editor-in-chief of SIAM’s Computational Science and Engineering series; Associate Editor of the SIAM Journal on Scientific Computing and editorial board member of seven other journals; member of the SIAM Program Committee; and member of the Science Steering Committee for the Computational Infrastructure for Geodynamics (CIG) project.

Juan C. Meza

Juan Meza is head of the High Performance Computing Research Department at E. O. Lawrence Berkeley National Laboratory. Dr. Meza oversees work in computational science and mathematics, computer science and future technologies, scientific data management, visualization, numerical algorithms and application development. He is responsible for developing short- and long-term research and development plans and proposing new technology directions. His current research interests include nonlinear optimization with an emphasis on methods for parallel computing. He has also worked on various scientific and engineering applications including scalable methods for nanoscience , power grid reliability, molecular conformation problems, optimal design of chemical vapor deposition furnaces, and semiconductor device modeling. Prior to joining Lawrence Berkeley National Laboratory, Dr. Meza held the position of Distinguished Member of the Technical Staff at Sandia National Laboratories and served as the manager of the Computational Sciences and Mathematics Research department. In this capacity, he acted as the Research Foundation Network Research program manager, the ASCI Problem Solving Environment Advanced Software Development Environment program manager and served as a member of the Sandia California site Research Council. Dr. Meza was recently named to Hispanic Business Magazine’s Top 100 Influentials in the area of science. In addition, he was elected a Fellow of the American Association for the Advancement of Science. In 2008, Dr. Meza was the recipient of the Blackwell-Tapia Prize and the SACNAS Distinguished Scientist Award. He was also a member of the team that won the 2008 ACM Gordon Bell Award for Algorithm Innovation. Dr. Meza has served on numerous external committees including, the Department of Energy’s Advanced Scientific Computing Advisory Committee, the Mathematical Sciences Research Institute’s Human Resources Advisory Committee, the Institute for Pure and Applied Mathematics Board of Trustees, the Institute for Mathematics and its Applications Board of Governors, SIAM Board of Trustees, and NPACI’s External Advisory Committee.

Eric Michielssen

Eric Michielssen is Professor of Electrical and Computer Engineering at the University of Illinois at Urbana-Champaign. His research interests include all aspects of theoretical and applied computational electromagnetics with an emphasis on the development of fast frequency and time domain integral-equation-based techniques for analyzing electromagnetic phenomena and robust optimizers for electromagnetic/optical devices. He is the (co-)author of 120 journal articles and book chapters and 180 conference papers and abstracts. He was the recipient of a 1994 International Union of Radio Scientists (URSI) Young Scientist Fellowship, a 1995 National Science Foundation CAREER Award, and the 1998 Applied Computational Electromagnetics Society Valued Service Award. In addition, he was named 1999 URSI United States National Committee Henry G. Booker Fellow and selected as the recipient of the 1999 URSI Koga Gold Medal. Recently, he was awarded the UIUC’s 2001 Xerox Award for Faculty Research, and appointed Beckman Fellow in its Center for Advanced Studies, UIUC Scholar, and Sony Faculty Fellow. He is an Associate Editor for the IEEE Transactions on Antennas and Propagation and a Fellow of the IEEE.

Vijayan N. Nair

Vijayan N. Nair is the Donald A. Darling Professor of Statistics and Professor of Industrial and Operations Engineering at the University of Michigan, Ann Arbor. He has been chair of the Department of Statistics since 1998. He was a research scientist in the Mathematical Sciences and Operations Research Centers at Bell Laboratories from 1978-1993. His research interests include engineering statistics, design and analysis of experiments, reliability engineering, and process control. He is involved with the Center for Radiative Shock Hydrodynamics (CRASH) at the University of Michigan, one of five national centers funded under Predictive Science Academic Alliance Program by the National Nuclear Security Administration’s Office of Advanced Simulation and Computing. As part of this center, Dr. Nair has been involved in modeling and analyzing data from large-scale simulation models and in uncertainty quantification. He has extensive practical experience in the automotive, semiconductor, and telecommunications industries. He is a Fellow of the American Association for the Advancement of Science, the American Statistical Association, the American Society for Quality, and the Institute of Mathematical Statistics. He is Vice President of the International Statistical Institute, President-elect of the International Society for Business and industrial Statistics, and Chair of the Statistics Division of the American Society for Quality.

Charles W. Nakhleh

Charles W. Nakhleh is a staff member in the Inertial Confinement Fusion Target Design Department (1684) at Sandia National Laboratories since December, 2007. From 2005 to 2007, he was the acting Group Leader and Deputy Group Leader for the Thermonuclear Applications Group (X-2) of the Los Alamos National Laboratory, where, among other tasks, he oversaw the W88 and Reliable Replacement Warhead efforts. He has also served as the Project Leader for the Quantification of Margins and Uncertainties (QMU) Tools and Methods project. In 2003, he spent a year in the Statistical Sciences Group at Los Alamos conducting research on the application of Bayesian uncertainty quantification techniques to simulation predictions. He joined X-2 as a staff member in 1998, and he served as the point of contact for the W80 secondary. He was a member of study teams that received Department of Energy Awards of Excellence in 1999, 2000, 2005, and 2007. He has served on a wide variety of advisory panels, including the NNSA’s Predictive Science Panel, the Los Alamos Director’s advisory panel on weapons certification, and as a consultant to the 2009 JASON study on warhead Life Extension Programs. His research interests span a wide range of ICF, weapons physics, and applications of Bayesian inference techniques. Before joining X-2, he was a staff member in the Safeguards Systems Group (NIS-7), where he worked on a variety of programs related to nuclear arms control and nonproliferation, including the application of Bayesian inference techniques to problems of verification. He received his PhD in Physics from Cornell University in 1996.

Douglas Nychka

Douglas Nychka is Director of the Institute of Mathematics Applied to Geosciences at the National Center for Atmospheric Research (NCAR), a interdisciplinary component with a focus on transferring innovative mathematical and statistical tools to the Geosciences. Dr. Nychka is a statistical scientist with an interest in the problems posed by the analysis of geophysical data sets. His Ph. D. (1983) is from the University of Wisconsin and he subsequently spent 14 years as a faculty member at North Carolina State University. His interest in environmental problems and a background in fitting curves and surface to spatial data lead him to assume leadership of the statistics project at the National Center for Atmospheric Research (NCAR).

Stephen M. Pollock

Stephen M. Pollock (NAE) is retired as Herrick Professor of Manufacturing and Professor of Industrial and Operations Engineering at the University of Michigan. He taught courses in decision analysis, mathematical modeling, dynamic programming, and stochastic processes. His research activities include developing cost-optimal monitoring and maintenance policies, sequential hypothesis testing, modeling large multi-server systems, and dynamic optimization of radiation treatment plans. He was the Director of the Program in Financial Engineering and the Engineering Global Leadership honors program. He has been Area Editor of Operations Research, Senior Editor of IIE Transactions, President (1986) of the Operations Research Society of America, and a Senior Fellow of The University of Michigan Society of Fellows. Pollock is a founding Fellow of the Institute for Operations Research and the Management Sciences, and was awarded its Kimball Medal in 2002. He was a member of the Army Science Board and is a member of the NAE.

Howard A. Stone

Howard A. Stone (NAE) is a professor in the Department of Mechanical and Aerospace Engineering at Princeton University. He received the Bachelor of Science degree in Chemical Engineering from the University of California at Davis in 1982 and the PhD in Chemical Engineering from Caltech in 1988. Following a postdoctoral year in the Department of Applied Mathematics and Theoretical Physics at the University of Cambridge, in 1989 he joined the faculty of the (now) School of Engineering and Applied Sciences at Harvard University, where he eventually became the Vicky Joseph Professor of Engineering and Applied Mathematics. In 1994 he received both the Joseph R. Levenson Memorial Award and the Phi Beta Kappa teaching Prize, which are the only two teaching awards given to faculty in Harvard College. In 2000 he was named a Harvard College Professor for his contributions to undergraduate education. Recently he moved to Princeton University where he is Donald R. Dixon ’69 and Elizabeth W. Dixon Professor in the Department of Mechanical and Aerospace Engineering. Professor Stone’s research interests are in fluid dynamics, especially as they arise in research and applications at the interfaces of engineering, chemistry and physics. His group tackles problems with a combination of experimental, theoretical, and modeling approaches. He has received the NSF Presidential Young Investigator Award, is a Fellow of the American Physical Society (APS), and is past Chair of the Division of Fluid Dynamics of the APS. For ten years he served as an Associate Editor for the Journal of Fluid Mechanics, and is currently on the editorial or advisory boards of New Journal of Physics, Soft Matter and Physics of Fluids. He is the first recipient of the G.K. Batchelor Prize in Fluid Dynamics, which was awarded in August 2008. In 2009 he was elected to the National Academy of Engineering.

Alyson G. Wilson

Alyson G. Wilson is Associate Professor of Statistics at Iowa State University. Until 2008, she was a technical staff member and the Technical Lead for DoD Programs in the Statistical Sciences Group at Los Alamos National Laboratory. Prior to her move to Los Alamos, Dr. Wilson was a senior operations research systems analyst working in support of the U.S. Army Operational Evaluation Command, Air Defense Artillery Evaluation Directorate. She also spent two years at the National Institutes of Health performing research in the biomedical sciences. Her research focuses on Bayesian methods, with emphasis on reliability modeling and information combination. She is currently the chair-elect of the American Statistical Association Section on Statistics in Defense and National Security and the chair of the ASA President's Task Force in Defense and Security. She received her Ph.D. in Statistics from the Institute of Statistics and Decision Sciences at Duke University.

Michael R. Zika

Michael R. Zika is the KULL Project Leader and a Group Leader at Lawrence Livermore National Laboratory. He earned his B.S. from Purdue University in 1991, his M.S. from Purdue University in 1992, and his Ph.D. from Texas A&M University in 1997, all in nuclear engineering. In 1997 he joined Lawrence Livermore National Laboratory as a computational physicist. His work focused on algorithms and physics models for modeling radiative transfer. As KULL Project Leader he has led a large team of computational physicists and computer scientists to deliver massively parallel, 2D/3D multi-physics simulation tools for the Stockpile Stewardship Program that have been used to design and analyze experiments on the National Ignition Facility. He has served as adjunct faculty at Texas A&M University and visiting faculty at University of California, Berkeley. He has participated in a variety of strategic planning efforts at the request of the ASC Program Office in DOE/NNSA.

Consensus

Read Online

Download PDF

Paperback - $37.80
Ebook - $30.59

Buy the book

Advances in computing hardware and algorithms have dramatically improved the ability to simulate complex processes computationally. Today's simulation capabilities offer the prospect of addressing questions that in the past could be addressed only by resource-intensive experimentation, if at all. Assessing the Reliability of Complex Models recognizes the ubiquity of uncertainty in computational estimates of reality and the necessity for its quantification.

As computational science and engineering have matured, the process of quantifying or bounding uncertainties in a computational estimate of a physical quality of interest has evolved into a small set of interdependent tasks: verification, validation, and uncertainty of quantification (VVUQ). In recognition of the increasing importance of computational simulation and the increasing need to assess uncertainties in computational results, the National Research Council was asked to study the mathematical foundations of VVUQ and to recommend steps that will ultimately lead to improved processes.

Assessing the Reliability of Complex Models discusses changes in education of professionals and dissemination of information that should enhance the ability of future VVUQ practitioners to improve and properly apply VVUQ methodologies to difficult problems, enhance the ability of VVUQ customers to understand VVUQ results and use them to make informed decisions, and enhance the ability of all VVUQ stakeholders to communicate with each other. This report is an essential resource for all decision and policy makers in the field, students, stakeholders, UQ experts, and VVUQ educators and practitioners.

Publication information

144 pages · 8.5 x 11 · paperback
ISBN Paperback: 0-309-25634-8
ISBN Ebook: 0-309-25635-6
DOI: https://doi.org/10.17226/13395

Suggested citation

National Research Council. 2012. Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification. Washington, DC: The National Academies Press.

Import this citation to:

Log In

Sign in to access your saved publications, downloads, and email preferences.

Former MyNAP users: You'll need to reset your password on your first login to MyAcademies. Click "Forgot password" below to receive a reset link via email. Having trouble? Visit our FAQ page to contact support.

Members of the National Academy of Sciences, National Academy of Engineering, or National Academy of Medicine should log in through their respective Academy portals.

Email (required)
Password (required)

Don't have an account?

Register

Email (required)
New password (required)
Retype password (required) Password must be at least 8 characters
I accept the terms of use
terms of use

Thank You

Thank you for creating a MyAcademies account!

Enjoy free access to thousands of National Academies' publications, a 10% discount off every purchase, and build your personal library.

Forgot Password

Enter the email address for your MyAcademies (formerly MyNAP) account to receive password reset instructions.

Reset Requested

We sent password reset instructions to your email . Follow the link in that email to create a new password. Didn't receive it? Check your spam folder or contact us for assistance.

Reset Password

New password (required)
Retype password (required) Password must be at least 8 characters

Reset Password

Your password has been reset.

Log In

Verify email

Verify Your Email Address

We sent a verification link to your email. Please check your inbox (and spam folder) and follow the link to verify your email address. If you did not receive the email, you can request a new verification link below

Subscribe

First Name
Last Name
Email (required)
I agree to receive emails from the National Academies (required)

We respect your privacy and will never share your information.

Mathematical Science Foundations of Verification, Validation, and Uncertainty Quantification

Featured publication

Description

Collaborators

Committee

Sponsors

Staff

Major units and sub-units

Contact

Provide feedback

Request materials

More like this

Review of the draft United by Nature assessment

Modernizing Mathematics Education for Grades 9-14

Health Care Financing to Improve Availability of Affordable Contraception: Identifying Promising Practices

Transit Cooperative Research Program

National Cooperative Highway Research Program

Prevention of and Response to Sexual Harassment in Higher Education: Tools for Department Chairs and Principal Investigators: A Workshop

Advances in Multimodal Artificial Intelligence to Enhance Environmental and Biomedical Data Integration-A Workshop

AI to Assist Mathematical Reasoning: A Workshop

Utilizing Advanced Environmental Health and Geospatial Data and Technologies to Inform Community Investment

Publication information

Suggested citation

Publication information

Suggested citation

My Academies

Mathematical Science Foundations of Verification, Validation, and Uncertainty Quantification

Dropdown items

Featured publication