10

Measurements and Calibration for Statistics, Data Analytics, and Artificial Intelligence

Planning committee member Ralph G. Nuzzo, University of Illinois at Urbana-Champaign, moderated a panel examining measurement and calibration issues.

THE ADDITIVE MANUFACTURING BENCHMARK SERIES

Lyle Levine, National Institute of Standards and Technology, began his presentation by talking about the Additive Manufacturing Benchmark Test Series (AM-Bench), which was instigated by a 2015 National Academies of Sciences, Engineering, and Medicine workshop. AM-Bench is “a continuing series of highly controlled benchmark measurements for additive manufacturing, with modeling challenge problems and a corresponding conference series.” The goal, he said, is to provide “really rigorous measurement data that modelers can use to validate their measurements,” and AM-Bench does this by providing sets of benchmarks that each cover a specific application space. For example, its benchmark for laser powder bed fusion (LPBF) has information on feedstock characteristics, build characteristics, in situ measurements, residual elastic strains, distortion, mechanical behavior, and microstructure before, during, and after heat treatment.

Levine offered a collection of best practices when using measurement data such as those supplied by AM-Bench. First, never take measurements at face value, as they have uncertainties and may have errors as well; even data from the same machine may not be comparable. Calibration of

both the build machine and the instruments is critical for transferability. In situ, high-rate data are often the most suitable for artificial intelligence (AI) and machine learning (ML) applications, but they are prone to large measurement uncertainties. Unambiguous scan paths and time–space data registration for sensors are needed for modeling. Features derived from ML should reflect physically interpretable metrics. Also, training and output data should be kept as close as possible to the measurements; the use of model-derived quantities should be limited.

He next described the Additive Manufacturing Metrology Testbed, which the group uses to do most of its LPBF builds on. This is a metrology platform that does builds rather than a build machine that does measurements, he said. It can produce multiple synced data streams that produce a detailed, multidimensional record of, say, the movement of the laser over the powder.

He closed with a list of data and calibration ML concerns from the perspective of qualification and certification.

SETTING UP WORKFLOWS FOR ADDITIVE MANUFACTURING

Keerti Kappagantula, Pacific Northwest National Laboratory, spoke of her experience in setting up workflows for advanced manufacturing, including AM. She began by listing some of the advantages of advanced manufacturing and noting that there is a critical need in the field to reduce the time for development. The reason it takes so long to develop and deploy an advanced technology, she said, is that there are so many factors that influence the performance of the components that are manufactured, and this is where AI can be valuable—by discovering which combinations of factors lead to optimal performance and pointing toward an effective workflow.

To illustrate, Kappagantula offered two examples of how her team helped set up workflows for the advanced manufacturing of parts. The first involved the production of materials for use in high-temperature environments; in that case, one lesson was the importance of consciously making parts that were not acceptable in order to better study the factors that influence good versus bad performance. The second involved infusing nanoparticles into ceramics to reduce porosity. In that case, the original plan was to observe the parts with computed tomography (CT) scanning, but when the small manufacturer said that it could not afford to procure a CT machine, Kappagantula’s group came up with a surrogate that could correlate the process conditions with the resulting material properties.

She concluded by listing a number of data considerations for workflow development. To begin with, how much data is “enough data”?

What does “good data” mean for generators of data versus users of data? How does one take into account system calibration and uncertainty in measurement versus experimental standard error? And in interpreting results, how does one differentiate between causation and correlation?

HANDLING DATA FOR USE BY ARTIFICIAL INTELLIGENCE

Echoing other speakers, Laura Bruckman, Case Western Reserve University, also emphasized the importance of transforming data into knowledge and using that knowledge to inform an AM workflow, but her specific focus was on developing ways to handle data so that these data can be used effectively by AI and ML programs.

To this end, Bruckman’s group created the Common Research Analytics and Data Lifecycle Environment (CRADLE), which is able to handle and analyze huge amounts of time-series, geospatial, and high-speed camera data. Its first use was with solar energy data. CRADLE takes advantage of distributed resources to increase its computing power in a cost-effective way and uses a variety of open-source technologies to manage heterogeneous data at petabyte scale. In particular, the group focused on being able to handle the data before it ever began working with AI models.

Bruckman emphasized the importance of lowering barriers and making such computing resources widely available. She also said that AI-ready data should follow the FAIR principles—that is, it should be findable (machine readable with metadata), accessible (data and metadata understandable to humans and machines), interoperable, and reusable—and that ontologies should be developed to provide structure for standards.

By handling data in this way, she said, it will become possible to move beyond human-centered investigations to data-centric AI investigations, which will dramatically increase the rate of scientific progress.

OPERANDO SYNCHROTRON EXPERIMENTS TO STUDY ADDITIVE MANUFACTURING

Tao Sun, Northwestern University, described how operando synchrotron experiments are being used to gain greater understanding of the physics underlying AM processes, such as the behavior of melt pools in LPBF (Figure 10-1). The experiments are carried out at synchrotron facilities, such as the Advanced Photon Source (APS) at Argonne National Laboratory, with the imaging being carried out while the process of interest, such as the laser melting of particles in a bed, is occurring. This provides in-process data that make it possible to develop detailed physics-based models of the activity.

The sorts of data that are provided by operando synchrotron studies are particularly useful in studying LPBF, Sun said, because it is a highly complex and dynamic phenomenon, but also one that is key to understanding structure defects in materials produced by LPBF and achieving repeatability and reliability of the process.

Operando synchrotron X-ray techniques offer a number of unique capabilities, he said, including probing subsurface structure dynamics, direct visualization of defect generation, and the ability to quantify important structural and process parameters.

After listing some of the AM-related operando synchrotron experiments being performed at APS and in Europe, Sun went into detail on how the experiments can facilitate understanding of the fundamental physics underlying AM processes, such as melt flow, cooling rate, and particle spattering, which in turn helps to inform, calibrate, and validate physical models. He closed by talking about the next phase of synchrotron experiments.

DISCUSSION

The first question concerned ways to improve throughput in AM. Levine said that by cross-calibrating machines and sensors, it is possible to improve control of the process, reduce the number of failures, and increase throughput. Bruckman said that by having a better understanding of variance and thus being able to determine in real time how parts are deviating from specifications, it is possible to determine sooner when the system is drifting too far from requirements and thus lose less time to wasted efforts and improve throughput. Kappagantula said that AI can significantly help to improve real-time predictive control systems, which can make it possible to maintain higher quality for longer periods of time, which, again, would improve throughput.

Levine was asked whether, with the rapid improvement in AM technologies, it is difficult to maintain the relevance of the testbeds. It is absolutely challenging, he said. The AM testbed that his group uses is not a static structure but rather is constantly evolving, with new sensors being deployed and new sensor systems being tested. Trying to stay abreast of the current state of the art is a never-ending challenge, he said.

In response to another question, Levine said that the development of standards always lags the need for standards. The development of standards is an expensive, time-consuming process and is not something that is undertaken lightly. He called for any interested audience members to get involved and to offer their opinions to standard-setting organizations on which new standards are needed.

Addressing the issue of simplifying some of the complexity found in AM, Levine noted that in nature there are highly complex systems that produce simple outcomes for various reasons—spatial averaging, for instance, or self-organization. Analogously, he said, not all of the parameters in AM systems may be necessary for understanding the outcomes. For instance, understanding a material produced by AM usually comes down to how rapidly the material is solidifying and the thermal gradients in the material during solidification, and from those it is possible to have a pretty good idea of, for example, what the solidification microstructure will be. So if it were possible to get a decent thermal model of how these things are produced, it would go about 80 percent of the way toward knowing what the phases and structures of the produced material would be. Similarly, there are relatively simple models that can help predict grain growth. In short, not all of the complexity is necessary, but one needs a good enough understanding of the process to know what needs to be included and what can be neglected. “You can get 80 or 90 percent of the way there by throwing away 80 or 90 percent of the complexity,” he said.

In response to another question, Sun said that while it is difficult and time-consuming to transfer knowledge into practice, there should still be people working on understanding everything that is associated with the process, at least to provide a knowledge base. A good way to transfer the fundamental knowledge that researchers develop is to build better models, he said. Physics understanding can also be assimilated into ML models. However, in terms of using physics knowledge to improve AM processes in a manufacturing facility, there is still a long way to go.

Concerning a question about what metadata should be collected, particularly as related to AI uses, Bruckman said that in addition to information on the particular code version being used for calculations, it is also crucial to keep records of what data a model was trained on. This is particularly important for repeatability. Kappagantula added that as an experimentalist, she thinks about metadata as everything that is involved in generating a data point, such as settings on sensors. “Record everything,” she said. “It is very important.” In physics-based models, Levine added, there are many different types of metadata that should be recorded: boundary conditions, input variables, input parameters, and so on. “Essentially there needs to be enough information that someone else could reproduce that simulation and get a comparable result.”

Concerning the differences between small-lot and large-lot production, Kappagantula said that calibration becomes much more important when one is making a large number of the same product, and quality assurance and reproducibility also become important. Levine added that if one is making a few parts rather than a thousand, it is possible to do qualification at the part level rather than at the process level.

On the topic of which aspects of physics and chemistry are not adequately captured in today’s process models, Sun said that there are many. For instance, there is not a good model for tracking powder motion. Shockwaves in melt pools and plasma generation are other examples.

The panel closed with a discussion of the importance of sharing data among organizations and the roles that standards and metadata play in that sharing. There was general agreement that gathering comprehensive metadata is crucial but also that it is difficult to get experimentalists to make the necessary effort to do that. Sun commented that having a better physical understanding of the phenomena helps in understanding which data are most important to collect.