metrology laboratory. Millipore/Sigma is one of the largest for-profit certified reference material producers (Our relationship prior to 2020 was limited to distribution of our products through their supply chain. Currently (FY 2022) nearly 21% of our sales are handled by these two businesses, up from 11% approximately 7 years ago. As of 2020, ORM has been in discussions to increase our utilization of the capabilities of each of these two CRM producers. We are providing them with advanced marketing materials for new and renewal products and advanced information regarding products in our pipeline.

We’ve met with LGC and Millipore leadership to discuss expanding our relationships. Given that both have significant manufacturing, packaging and certification capabilities as well as secure supply chains (from warehouse to customer) it behooves ORM to learn and partner with these entities. Currently NIST (ORM) makes no distinction between selling to an end user, a licensed distributor, or a reseller. This was never a problem for our “20th” century products, i.e., cements, metals, organic solutions and industrial materials. Now however our portfolio is evolving to include protein biologics, health and clinical materials, and cells, all of which require exquisite handling and storage to ensure the integrity of our certified reference materials. In the next FY we will be exploring the practice of pre-positioning NIST SRM products in accredited warehouses for both LGC and Millipore to ensure adequate care of our products for our foreign customers. We anticipate prepositioning product in both the EU (Germany) and Asia (Singapore). In addition, the ORM director is in discussion with both regarding advanced packaging, processing and distribution of NIST products, perhaps under a co-label. The ORM director is also engaged internally and externally to determine the feasibility of expanding the NIST Traceable Reference Materials (NTRM) beyond our SRM gas program. Implementation of this program could ensure adequate supply of our products to our stakeholders, expand upon the offerings of our SRM program, and relieve our laboratorians of time-consuming support work related to SRM support (i.e., stability monitoring, homogeneity assessment etc.).

Workshop on Digital NIST in September 2022.

Over 450 participants were in attendance at this virtual conference. Included were discussions on digital reference material certificates. Worked with industry, academia and government to gauge interest in developing a digital alternative to the current SRM paper and PDF certificates. A white paper was published and efforts are underway to implement these new certificates. This will be a much more technically challenging task than creating digital calibration certificates as SRMS have many more and different measurands and SRM certificates have a diversity of formats.

E-commerce.

ORM IT team implemented and transitioned all measurement services from custom in-house software business systems to the NIST store front. Now all measurement services are on one common platform serviced and aided by ORM’s implementation of Sales Force customer relationship management system.

Development of Research Grade Test Materials Concept.

Originally developed from a strategic plan eight years ago to fully realized in FY22 to improve evaluation of candidate materials for RM/SRM development. ORM is currently developing the infrastructure to assist in all phases of implementation of the RGTM process to include support for SDS authoring, shipping and packaging expertise, and ecommerce implementation (document retention) to support in excess of 20 interlaboratory studies per year. Expanding service to possibly all of NIST.

NTRM.

Started discussions with external CRM producers to support joint efforts on making and supporting RM’s that are out of stock and/or need external manufacturing capability.

Other NMIs.

Actively engaged with NMIA and JRC Geel to discuss business models and share best practices.

MML Response to Recommendation 3-2:

ORM has modified our process for evaluation of service development projects. We utilize the RGTM process that includes an interlaboratory study to gauge stakeholder interest and engagement and feedback on proposed projects. With regard to ongoing production projects, we do, in collaboration with the participating division, prioritize development and renewal of high demand SRM’s over less in-demand products, to maximize our ROI from the WCF.

MML Response to Recommendation 3-3:

In FY 22 the ORM director gave three internal talks on education and improvement for SD projects and the new RGTM process to MML specific and NIST general audiences. In FY 21 the ORM director gave four division level internal talks to discuss the “Guide to SRM Development” and the impact of several of our newer products. The ORM director and staff regularly meet one on one and in groups to discuss all aspects of measurement service development from origination of ideas to funding, to stakeholder engagement and outreach. Beginning in April of FY 23 ORM will host SD projects presentations to the NIST audience to encourage participation from those unfamiliar with SRMs and to share the very excellent science and metrology of the supported projects.

MML Response to Recommendation 4-1:

Since the 2020 NASEM assessment, MML has developed complementary efforts aimed at leveraging Data as an Asset, driven initially by elements of the MML strategic plan for Next Generation Data. These efforts included establishment of Laboratory Information Management Systems (LIMS), a data infrastructure development and deployment effort referred to as Data Plumbing, and work to support development of sound Artificial Intelligence (AI) and Machine Learning (ML) tools.

In 2020, led by the Office of Data and Informatics (ODI), MML assembled a team of representatives from across MML to establish a LIMS Community of Interest (COI). This team prioritized projects from each division, which were used to develop and prototype LIMS capabilities. These projects factored in lessons learned from early LIMS adopters such as MML’s NIST Center for Automotive Lightweighting. In recent years, this effort has produced several successful and hi-impact LIMS, including the NexusLIMS which serves electron microscopy facilities across MML and NIST. To spread LIMS further, core shared services and infrastructure and specialized tools for research were developed, supported in large part by the Data Plumbing effort, discussed next. These capabilities, and related best practices, were developed in consultation with research subject matter experts and Data Science experts from both within and outside of NIST. In 2021, the MML LIMS COI published a roadmap for MML that serves as a key guide to ongoing development of LIMS capacity.

The Data Plumbing effort has been highly successful establishing infrastructure needed to automate and orchestrate data movement from instruments laboratories, through local network storage to newly deployed centralized storage. In partnership with the NIST Office of Information Systems Management, MML purchased a modern EMC Isilon 1.25 Petabyte storage array for centralized storage of research data. This highly redundant, highly available storage ensures that MML’s valuable data is both protected and accessible. This new enterprise class storage array provides key capabilities for MML researchers to store and access large datasets from across the NIST network and to integrate data for machine learning and other advanced analyses. In addition, the Data Plumbing project has deployed infrastructure to automate data transfers from laboratory instruments to central storage. Data files are stored in a structure that identifies originating instrument, creation date, and in some cases instrument operator. Files are read-only to preserve data integrity and are broadly accessible to NIST researchers. Storing data in a secure, defined, broadly accessible location is seen as the foundation for all future data management efforts. To date, approximately 95 instruments have been “plumbed” for automated data transfer.

Deployment of AI/ML capabilities have been widespread across NIST, and research data is the critical component to the development of algorithms and validation of solutions. To leverage its own data, MML Established the MML AI/ML working group, which has identified data and code assets to benchmark and demonstrate AI/ML in the research mission space.

In addition, the MML ODI continues to work with the NIST Special Programs Office and NIST Library services to provide Data Management Planning tooling and Data Publication services critical to establishing data as an asset for the organization. More details are specified in Division 641 NASEM report chapter. It is notable the MML continues to lead efforts in critical exercises for data management planning, documentation and support across NIST. Exercises are underway to expand practice and use of planning tools, within the context of individual areas of research.

MML Response to Recommendation 4-2:

ODI has made greater use of detail assignments from other divisions in MML to broaden both our capacity and our footprint in the laboratory. We also have less formal liaison agreements with MML staff from other divisions who regularly participate in group and division meetings. Detailees and liaison staff have joined ODI from divisions 642, 643, 646, and 647, and we have worked closely with 642 on the AMBench platform and 644 on the deployment of LIMS for biosciences. We also have close collaborations with the NIST Library (data curation and publication, Research Data Framework).

MML Response to Finding:

Widely used methods for recruitment of postdocs at NIST are not amenable ODI. For example, the NRC postdoc competition does not focus on data science. We have made some efforts to build stronger connections with MML postdocs whose research relies on data science and AI/ML.

MML Response to Recommendation 4-3:

This is an excellent suggestion. Due to competing priorities, ODI has not yet been able to expand the software carpentry program. As noted in Response 4.2, ODI has expanded its efforts to interact with division staff in ways that include mentorship and other guidance on sound computational techniques.

MML Response to Recommendation 4-4:

Development of reference materials occurs in MML’s technical divisions, with the MML ORM coordinating seed funding, scale up, packaging, sales and distribution. Respectfully, is unclear what this recommendation suggests ODI or MML do or change with respect to reference materials development.

MML Response to Recommendation 5-1:

MSED operates the Center for Theoretical and Computational Materials Science (CTCMS), a high-performance computing (HPC) resource supporting computational research in MSED (and other MML divisions), including the efforts within the Thermodynamics and Kinetics Group. MSED makes annual investments of approximately $200K that largely maintain current capabilities as computing hardware technologies advance. MSED similarly provides approximately $100K annually to support another HPC resource at NIST (Raritan, operated by OISM on behalf of participating labs) in addition to the annual use-based charges (≈$150K). Beyond these annual investments, there have been a few recent targeted investments that support specific use cases / projects in MSED and in other MML divisions, some of which have increased CTCMS capabilities to provide value to other users (e.g., GPU resources, additional nodes and storage). Additionally, many of the MGI-supported efforts in MSED involve data repositories that depend on the NIST Configurable Data Curation System (CDCS) as their data infrastructure, such that MSED invests in contract support for the associated CDCS instances. MSED does not currently have the resources to invest beyond these largely maintenance level support costs. Strategic investment and associated NIST-level commitment is needed (and is the subject of NIST HPC planning efforts) to support the broader NIST community in this regard.

These limited investments reflect the current resource constrained environment and an effort to balance investment needs across our portfolio, including experimental capabilities that support empirical approaches. Recent examples specific to the Thermodynamics and Kinetics Group

include a gas-cooled vacuum furnace (replaces 20+ year old system, procurement underway), a lab-scale directed energy deposition (DED) additive manufacturing capability (procurements underway), and electron microscopy upgrades that included a broad-beam ion mill and new XEDS/EBSD detectors for our JEOL 7100 SEM (procurements complete).

MML Response to Recommendation 5-2:

MSED has developed a new strategic plan. Strategic planning activities commenced in FY2021 following the 2020 NASEM review, culminating in a living strategic plan that is being used to prioritize investments during this period of intense resource constraints. This plan includes prioritization of scientific staff / expertise and investments in equipment, computational / data resources, and associated facilities. Three strategic goals were established to support state-of-the-art core competencies that provide unique value and impacts for our stakeholders and for addressing national initiatives (Goal 1), to modernize and sustain critical measurement services and standards efforts (Goal 2), and to expand capabilities in lab automation, high-throughput computation, and AI/ML methods to seed new capabilities in support of specific stakeholder needs in data and informatics (Goal 3). These goals support NIST Strategic Goal 1: Position NIST to Advance U.S. Science and Innovation, NIST Strategic Goal 2: Maximize NIST’s Stakeholder Impact through High-Value Service Delivery as well as several current Administration initiatives. In terms of the 2015 MML Strategic Plan, MSED Goal 1 aligns with MML Goal 1: Measurement Science Excellence; MSED Goal 2 aligns with MML Goal 2: Measurement Service Excellence; and MSED Goal 3 aligns with MML Goal 3: Data Science and Data Management Capabilities. Additionally, MSED Goal 1 supports MML Goal 4: Strategic Partnering and Customer Engagement.

MML Response to Recommendation 5-3:

MML to respond and may consider including the following information. Due to continued resource limitations brought on by increasing labor costs, overhead rates, renovation costs, equipment maintenance costs, etc., MSED has chosen to reduce its division overhead rate to a level just above the minimum required based on the Invested Equipment (IE) loan model that is part of the NIST WCF. That minimum level supports only a very limited set of annual equipment purchases and is often needed to upgrade or replace aging equipment across the division. Likewise, MML’s IE allocation is very limited relative to the needs across the lab.

MML Leadership has helped influence NIST-level changes to WCF / OH policies / management of high-dollar (shared) equipment needed to enable some new equipment purchases. While more movement is needed from NIST, MML was successful in arguing that purchases made with other agency funds be treated to a much lower OH rate.

MML Response to Recommendation 5-4:

MSED has limited resources and capabilities for using multiple forms of media to communicate research achievements. We have focused efforts on revamping our external website over the past year, working within the constraints of Drupal, which governs the available webpage formats. MSED has also partnered on several occasions with NIST PAO to highlight division research efforts. Links to recent examples are found on our external webpage and include:

• Spotlight: Mimicking Our Own Senses to Detect Ovarian Cancer
• Spotlight: Blood Vessels on a Chip for Dental Research and More
• With Fuzzy Nanoparticles, Researchers Reveal a Way to Design Tougher Ballistic Materials
• Spotlight: The Behavior of Polymers in Our Recycled Plastics
• Thermal MagIC: New NIST Project to Build Nano-Thermometers Could Revolutionize Temperature Imaging

In-person, virtual, and hybrid workshops and meetings have been used to engage with customers and collaborators over the last several years.

MML Response to Recommendation 6-1:

MMSD has a close relationship with the Office of Reference Materials (ORM), who manages the inventory and sales of new physical reference materials. For example, the MMSD Division Quality Manager reviews all customer inquiries received by ORM [at https://shop.nist.gov] to determine areas where MMSD should explore new reference materials projects and/or consider reissuing discontinued reference materials due to customer demand. This sharing of inquiry data is automated through ORM’s purchase of a Salesforce CRM (customer relationship management) software. To this end, MMSD is planning a workshop in 2023 aimed at obtaining feedback from the X-ray user community on the needs and uses of various Standards Reference Materials. MMSD will use this information to inform future production, staffing levels, and timely availability of products to the community.

MML Response to Recommendation 6-2:

In 2021, MMSD began a reorganization process to better support the programmatic focus areas. Researchers working on the X-ray Metrology project are now a part of the ‘Nanomechanical Properties Group’ where they continue to collaborate and coordinate the production of X-ray Metrology related SRMs. The Data and AI-Driven Materials Science Group (formerly known as the Materials for Energy and Sustainable Development Group) has realigned to conduct research, develop, and disseminates measurement science, data, models, workflows, and technologies pertaining to AI-driven materials science and data infrastructure operations related to materials and measurement science.

MML Response to Recommendation 6-3:

Since 2020, MMSD has identified six focus areas representing the division programmatic priorities. The focus area goals leverage core competencies across groups thereby promoting the assembly of diverse and agile research teams within the division.

Further recognizing the importance of supporting intra-division collaboration, the division has begun removing operational and/or cultural barriers to collaborative science. As a catalyst for more impactful intra-division collaboration, the division has had numerous cross-group/division-wide discussions and technical “showcases” targeted our priority programs towards elevating all staffs’ awareness of resources and technical efforts within the division.

MML Response to Recommendation 6-4:

The Division is constantly evaluating the impact of staff numbers on programmatic focus and impact. We have recently adopted a “single point of failure” analysis for several projects, which involves understanding what research efforts or relationships would be substantially impaired should a single staff member cease their activity, and continually evaluate staff motivation, engagement, and growth through the life of a given project. Since the beginning of the global pandemic (early 2020), the Division Office has held Division-wide meetings twice a month to communicate the Division’s mission, purpose, core competencies, and focus areas. The Division also strives for transparency about new hires, promotions, and equipment upgrades/purchases to help reinforce communication about Division priorities.

MML Response to Recommendation 6-5:

Hiring:

Since 2020, MMSD has implemented MML’s best hiring practices aimed at increasing transparency and fairness in the hiring process thus promoting a more diverse pool of qualified applicants. Two management vacancy positions (Deputy Division Chief and Group Leader) were competed, attracting strong diverse candidates. The selection process included input from diverse sources such as management, technical, and administrative staff. This process resulted in two female applicants being identified as best qualified for the needs of the organization and now serve in the Division management team.

Promotions:

Since 2020, three female scientists were promoted to ZP V, the highest pay band level for NIST in recognition of their outstanding technical leadership and impact in their field.

Recruitment:

MMSD launched the MMSD Academic Showcase Seminar aimed at inviting guest academic speakers to exchange ideas, identify areas of potential collaboration, and promote MMSD’s programs to future NRC/post-doctoral candidates. Furthermore, MMSD now utilizes LinkedIn to promote the division and its staff scientific achievements to a broad audience and LinkedIn Recruiter as a tool to attract a diverse applicant pool to opportunities within the division.

Retain:

MMSD has recommitted its internal efforts to have an engaged and inclusive work environment. In 2022, MMSD held a ‘People’ off-site where all staff, including associates and post-docs were invited to participate, provide feedback, and develop a strategic plan on how to improve the division. Initiatives such as improved onboarding for incoming staff, internal events that promote staff engagement and launching an informal mentoring group for early career staff are being led by two internal working groups.

MML Response to Recommendation 6-6:

The Materials Measurement Science Division is continually assessing all resources, including equipment, instrumentation, facilities, and personnel. There is a formal assessment at the beginning of each fiscal year, with a subsequent prioritization and process for funding consideration. Prioritization is based on many factors, including, but not limited to, strengthening the Division’s core competencies, growing programmatic focus areas, and succession planning. Where possible, the Division attempts to leverage the servicing of existing and purchase of new instrumentation and throughout both the Division, MML, and NIST. For example, the Division shares several electron microscopy service contracts with other MML Divisions and another Laboratory. More recently, the Division has purchased several new instruments through a cost-sharing relationship with other Divisions and the MML Lab Office. Such equipment purchases would have otherwise proved insurmountable for our Division alone, but MML staff and management are always seeking ways to collaborate, and the MML Lab Office has remained quite supportive for our programmatic needs.

expertise to the NIST Manufacturing Extension Program (MEP) program, a program dedicated to serving small and medium-sized businesses.

MML Response to Recommendation 7-2:

Our multidisciplinary team of scientists and engineers develop foundational measurement science and capabilities that serve a number of applications and different sectors of the bioeconomy. Our programs are designed to be agile and address the most pressing needs in various emerging industrial applications. For example, we have recently recruited a postdoc to our Engineering Biology Program, who brings expertise in culturing environmental microbes, such as those found in soil. She is transferring protocols and knowledge on measuring differential gene expression in microbes exposed to environmental pollutants, through RNA sequencing including the bioinformatics pipeline for the associated data. Another recent postdoc recruit brings expertise in yeast microbiology, especially engineering GPCR protein sensors for therapeutic applications. He also brings experience with ML/AI tools for protein engineering and library design for directed evolution. The NIST Genome Editing Consortium is focused on addressing the measurements and standards needed to increase confidence and lower the risk of utilizing genome editing technologies in research and commercial products mainly in the advanced therapies sector. However, the measurements and standards developed are applicable to other sectors such as agriculture.

MML Response to Recommendation 7-3:

Since the 2020 Panel review, the BBD has established multiple measurement capabilities to support the National priority areas of Engineering Biology, Microbiome, and RMAT. Additional space is urgently needed to continue to grow these emerging programs, but our laboratory spaces in buildings 227 and 224 are at full capacity. To address our pressing space needs, we have been working closely with MML leadership to identify space currently occupied by the ADA in building 224. As of January 2023, MML has agreed to provide 5 additional laboratory modules to help alleviate our pressing microbial program space needs in the short term. Access to additional space will also improve laboratory safety by reducing unnecessary exposure to potential hazards due to overcrowding and clutter (e.g., tripping hazards and storing heavy items high up) and providing sufficient bench space to safely work in the labs and ensure our laboratories are aligned with our Quality Management System. We continue to work with MML to initiate formal evaluation to determine whether the square footage assigned the BBD is commensurate with the division’s current size and mission.

MML Response to Recommendation 7-4:

Since the 2020 panel review, the BBD has established new programs in virology (e.g., viral and non-viral vectors), immunology (e.g., serology and neutralization assays for SARS-CoV-2), microbiology (e.g., Mpox Research Grade Test Material 10223), and bioinformatics (e.g., genomic cybersecurity, LIMS/data infrastructure for managing data/metadata in collaboration with

NASA/JPL). We were able to obtain additional funding to expand into adjacent disciplines related to material science and have strengthened partnership with other parts of NIST with deep domain expertise in thermodynamics, and colloidal science.

MML Response to Recommendation 7-5:

BBD is actively contributing to the broader NIST DEIA efforts. We consider underserved minority representation in our hiring practices. Through a partnership with Morgan State University, we provide paid internship opportunities to two graduate students. We have asked appropriate NIST offices with the proper authority to collect data related the BBD’s ethnic and diversity composition so that we can conduct analysis and develop appropriate action plans.

MML Response to Recommendation 7-6:

The BBD did not have any long-tenured postdoctoral fellows at the time of the 2020 review or during the current review period. The BBD benefits greatly from the contributions of our postdocs in terms of carrying out our mission, building alliances and bridges to other organizations, and mentoring students. We greatly support the career goals of each of our postdocs and in addition to the research and training opportunities that are available to them at NIST, they have the opportunity to meet and connect with NIST stakeholders through workshops, consortia, and other collaborative efforts that are relevant to their research and future career goals. For example, we have joined the Global Biofoundry Alliance (GBA), where we placed a postdoc, who was engineering protein sensors with our automated workflows. This provided the necessary experience for the postdoc and enabled him to take a position with the bio foundry at the Concordia Genome Foundry at Concordia University in Montreal, Canada.

MML Response to Recommendation 7-7:

We work to ensure that protocols important to our workflows are shared across the BBD (e.g., between technicians and from postdocs to technicians), to help guard against the loss of this knowledge in the Division. Our staff have also implemented electronic resources (e-notebooks, LIMS) that are shared across the groups and Division, to ensure access to protocols, data, and metadata associated with measurements and laboratory operations.

MML Response to Recommendation 8-1:

The BMD has taken several steps to address this recommendation. First, the BMD evaluated its portfolio of staff expertise and projects in AI/ML. It was found that the BMD’s activities in AI/ML fall into two broad categories: 1) the application of AI/ML approaches to advance biomolecular measurement science and tools, and 2) the development of reference materials and test data sets to advance AI/ML approaches applied to biomolecular measurement systems. Second, the BMD formed an AI/ML working group with a representative from each group to provide updates on AI/ML related projects in the division, exchange ideas, and to promote collaborations. A BMD staff member also represented MML on the NIST AI Community of Interest group, which helped the division to stay abreast of AI/ML activities and expertise across NIST and to raise the visibility of the BMD’s activities. Third, the BMD actively sought to leverage expertise and resources both inside and outside of NIST to expand its portfolio of AI/ML-related projects. Examples of new collaborative AI/ML-related projects are given below.

A new collaboration between scientists in the BMD’s Mass Spectrometry Data Center Group, the Biosystems and Biomaterials Division, and NIST’s Information Technology Laboratory (ITL) was initiated to use and develop artificial intelligence and other machine learning methods for a new class of genomics Standard Reference Materials and proteomics Standard Reference Data. The project was funded by ITL through a Use-Inspired proposal competition. The Mass Spectrometry Data Center Group developed neural networks to predict peptide spectra from their sequences and potential modifications. The models were trained on well-characterized NIST mass spectral libraries using over one million MS2 peptide spectra. The models were able to predict spectra and to evaluate over the collision energy range over which predictions are valid. One paper has been submitted on the topic of using AI predictions to improve mass spectral library searching. Two additional papers are in preparation, one on spectral prediction of proteins with post-translational modifications, and another on creating reliable mass spectral datasets for AI.

The Bioprocess Measurements Group collaborates with researchers from the University of Colorado, the FDA, and Ursa Analytics, Inc. in the development of convolutional neural networks (CNNs) to analyze flow imaging microscopic images of protein particles. The presence of protein particles in formulated protein therapeutics and their potential impact on safety and efficacy are of great concern to the biopharmaceutical industry and regulators. A resulting publication demonstrates that CNN analysis of microscopy images was able to determine distinguishing “fingerprint” features for different preparations and materials and that NIST particle reference materials can be used as a repeatable, quantitative control for image acquisition and analysis. The approach is promising for quality control applications such as the detection of manufacturing deviations (DOI: 10.1007/s11095-021-03130-9).

The Biomolecular Structure and Function Group collaborates with researchers from ITL, Brown University, George Mason University, and the University of Pennsylvania to develop ML methods for analyzing cryo-electron microscopy images of mRNA-Lipid Nanoparticle (LNP) structures. The structures of mRNA-LNP vaccines and therapeutics are complex containing a range of diameters, water-filled pockets of different sizes with RNA present or absent in cores or pockets. Cryo-EM is a promising tool for characterizing this heterogeneity; however, the tracing of large numbers of LNPs by hand is intractable and simple algorithms for finding particles perform poorly due to low contrast. Two approaches are being evaluated: 1) the training of ML models on manually-annotated micrographs using “Cellpose”, a deep-learning segmentation method designed for segmenting cells and 2) “U-net”, which uses simulated images.

The BMD machine learning portfolio also includes collaborations between NMR spectroscopy groups at IBBR, computational scientists in ITL, and collaborators at the Network for Advanced NMR, sited jointly at UCONN, UGA, and UW. Nearly all interpretation of biomolecular NMR in two dimensions and higher is performed by interactive inspection of 2D contour plots by experts, and identifying peaks is a required step in most every NMR workflow, both for biomolecular data and small molecules. The goal of this project has been to use machine learning and other methods to build a system that can identify and characterize peaks as effectively as a human analyst can. The method must be able to identify partially resolved peaks, discriminate true signals from artifacts, and make high-quality decisions about whether two peaks are related. In a first approach, a UNET convolutional neural network architecture has been used to identify peaks in contour plot images via segmentation. The project benefits from the nature of NMR data, which allows realistic training spectra to be constructed completely synthetically from mathematical models, or semi-synthetically, by combining individual signals from measured spectra.

MML Response to Recommendation 8-2:

Since the Panel’s recommendation in 2020, the BMD has initiated several projects in other biotechnology sector areas:

The development of LC/MS approaches to screen and identifying plastic degrading enzymes. This project was supported by a new NIST Circular Economy Initiative.

The development of a mass spectral library of plasticisers in high volume, commodity polymers with applications towards plastics recycling – a project also supported by the Circular Economy Initiative.

In collaboration with the Chemical Sciences Division, the development of tree nut reference materials — hazelnut and almond flours — that will help ensure accurate and consistent results for test kits that regulators and food manufacturers can use to detect tree nut allergens.

In direct response to this recommendation, the BMD has also taken several actions to develop a programmatic strategy to assess measurements and standards needs of other biotechnology sectors so that we are positioned for future collaborations, funding opportunities, and growth. First, we have been engaging stakeholders to learn and assess current and emerging measurement and standards needs in these other sectors. We have found BioMade, a Manufacturing Innovation Institute (MII) founded in Oct. 2020, to be a useful resource. BioMade’s goal is to advance sustainable and reliable bioindustrial manufacturing technologies for a wide array of products including chemicals, detergents, polymers, electronic films, fabrics, agricultural products (e.g., feedstock), crop protection solutions, food additives, fragrances, and flavors. Interactions with BioMade have been facilitated through the detail of a BMD staff member in NIST’s Advanced Manufacturing National Program Office which coordinates Manufacturing USA, the network of MIIs across the country that includes the biotechnology MIIs NIIMBL, BioMade and the ARMI/BioFab. Second, BMD management team and staff have also attended numerous conferences and workshops that have dealt with biotechnology applications for industrial chemistry, agricultural, energy, or climate including a recent Oct. 2022 NASEM workshop “Successes and Challenges in Biomanufacturing”. Third, we have played an active role in internal MML/NIST and interagency discussions on expanding the NIST biotechnology and biomanufacturing programs to advance key societal goals outlined by the Administration’s recent Executive Order “Advancing Biotechnology and Biomanufacturing Innovation for a Sustainable, Safe, and Secure American Bioeconomy”: (https://www.whitehouse.gov/briefing-

room/presidential-actions/2022/09/12/executive-order-on-advancing-biotechnology-and-biomanufacturing-innovation-for-a-sustainable-safe-and-secure-american-bioeconomy/).

Through these interactions and discussions we see the BMD and NIST playing important roles in providing measurement science and standards to support a wide array of emerging biomanufacturing technologies and products. Most biotechnology processes involve an upstream, cell culture operation where process analytical technologies (PAT) are used to generate real time data for monitoring and predicting process trajectories and ensuring the quality of products. Biomanufacturing facilities of the future will use PAT data combined with advanced, big data analytics including AI/ML to achieve consistent quality, higher yields, lower costs, and reduced waste – important goals for all bio-based production. Critical to the implementation of PAT will be the availability of robust measurement technologies with understood measurement uncertainty, the development of new measurement technologies, and platform standards to drive adoption of new manufacturing technologies. The BMD has initiated several efforts in upstream process measurement including the use of dielectric measurements for measuring live cell concentration, the use of NMR for on-line measurements of cell metabolites, and the use of LC/MS methods for at-line measurements of product attributes. These measurement approaches are general and applicable to any biomanufacturing process.

MML Response to Recommendation 9-1:

One of the strengths of the Chemical Sciences Division is the connection and interactions between members from different groups to provide metrological support to NIST stakeholders in the chemical sciences. The strength of these collaborations and synergies have significantly increased in recent years given the multidisciplinary nature of some of the grand challenge problems tackled by the Division. As an example, members from the Organic Chemical Metrology Group, Chemical Informatics Group, and the Biochemical and Exposure Science Group (in Charleston, SC) have been working together with scientists from the Engineering Laboratory in the development of a metrological framework aimed at the identification, characterization, and quantitation of PFAS in water, firefighter gear, sea food, and marine organisms.

MML Response to Recommendation 9-2:

For the most part, the Chemical Sciences Division does not have control over the allocation of NIST appropriated STRS funds. Once the STRS has been allocated to the Division, the proper

resources are dedicated in support of the programs and technical activities within the groups based on a diverse set of drivers, including stakeholder needs, NIST mission, MML mission, and Division mission. In addition, the research and measurement service delivery needs are also driven by the following focus areas:

• Develop and maintain state-of-the-art chemical measurement capabilities.
• Generate, gather, curate and disseminate critically evaluated data.
• Develop, implement and validate accurate theoretical models and reliable chemical informatic tools.
• Conduct research concerning the qualitative and quantitative determination of chemical composition.
• Provide measurement service delivery activities based on focus area needs.
• Disseminate tools for measurement traceability and quality assurance (such as reference materials and other services).
• Demonstrate international comparability of U.S. standards for chemical measurements.

In recent years, the Division has been working with MML management to delineate a plan leading to a more sustainable and forward-looking funding model to support the development of our SRM programs and to modernize the Division’s measurement services. NIST management has taken the suggestions of this plan and have included or are planning on including a line item in future NIST budgets to address these issues. Additionally, the Division management and the staff have been extremely active (and successful) at procuring additional funding from other agencies to support work in impactful programs that requires the technical excellence and expertise provided by the Division in Chemical Metrology.

MML Response to Recommendation 9-3:

The Division has appointed Kevin Huncik, from the Biochemical and Exposure Science Group, the Division Instrumentation Officer. One of his duties is to maintain a list of the instrumentation around the Division (including information regarding location, age, users, programs associated with the instruments, etc.), and advice the Division Chief in matters of repairs, maintenance, and acquisition of instruments.

MML Response to Recommendation 9-4:

In addition to exit interviews, CSD is actively keeping contact (whenever possible) with postdoctoral fellows that find employment outside NIST. In addition to help the Division obtain invaluable input regarding new opportunities, these interactions have also provided CSD a very efficient vehicle to reach out to other young scientists that might be willing to come and work in our division.

production costs of the Charpy SRM’s, this also covers the two ACMD staff who work full time on the Charpy program. To improve Charpy program customer service, the viability of using customer-provided electronic images of post-testing Charpy reference specimens was evaluated and a new online/e-mail based service was added to the Charpy verification program. This has been highly successful and was the basis for two ACMD staff receiving the NIST 2022 Judson C. French award for outstanding customer service by creating a digital verification program for Charpy standards involving >1900 customers per year in 60 countries. The resultant ever-increasing collection of post-test Charpy specimen images and associated data is now being used to develop an AI-based image evaluation tool with the expectation to be able to provide user-initiated on-demand evaluation and interpretation of Charpy specimen images. In general, such SRM capability development work is funded the NIST Standards Development (SD) program of the Office of Reference Materials. In short, both the current day operations and the forward-looking improvement of the Charpy SRM Program is self-financed. For the Thermodynamic Research Center SRD program, the annual revenue from SRD sales, subscriptions and consortium memberships is ~$600K, which covers roughly half the annual cost for providing these SRD services. Similarly, SRD sales of REFPROP bring in about $500K/year to ACMD which again covers about half the cost of producing, maintaining and distributing REFPROP. Both to potentially increase revenue and to serve more customers (stakeholders), the TRC is currently working with the Office of Data and Informatics to implement sales of data extractions (ie, customer-defined targeted subsets of the full TRC data and model collection) as there has been a dramatic increase in the demand for such a service due to researcher’s interest in well curated data sets for training AI models. ACMD is also hiring a full-time application programmer to modernize the front end of both TDE and REFPROP, which will both improve their utility for customers and enable new sales modalities aimed at broadening the customer base for these SRD products.

MML Response to Recommendation 10-3:

The mission of the Applied Chemicals and Materials Division (ACMD) is to characterize the properties and structures of industrially important fluids and materials, thereby providing a diverse stakeholder community with innovative measurements and models and dissemination of critically evaluated data, leading to improved processes and better products, as well as new and improved standards. This mission is very well aligned with the higher-level mission of NIST To promote U.S. innovation and industrial competitiveness by advancing measurement science, standards, and technology in ways that enhance economic security and improve our quality of life. “Applied” is part of the ACMD division name because of the ACMD focus on serving external stakeholders interests and needs in areas of chemical production including new greener chemical processes, reducing climate change through identification of alternate chemicals/processes, improving domestic manufacturing by advancing additive manufacturing, developing metrologies to advance the US semiconductor industry, developing robust breath analysis metrologies for public safety and human health and advancing metallurgy and metals metrology to enable meeting future infrastructure needs for new energy sources and/or mitigating climate change. While all but one ACMD staff member are located at NIST-Boulder, ACMD actively engages with NIST-Gaithersburg based staff and research groups to bring the best NIST resources and researchers together to address any specific challenge. At the present time, ACMD staff are engaged in 20 distinct cross-campus research collaborations of which 13 are not just cross-campus, but also involve collaborations with NIST staff outside of MML. One such project is a joint PML/MML effort on Semiconductor Gas Flow Metrology which is one of the five projects selected across NIST to receive FY22 funding as a NIST internal CHIPS Act quick start project. ACMD’s very productive and increasing engagement in collaborative research with NIST-Gaithersburg stems in part from a 2019.

ACMD Leadership decision to be more actively engaged with NIST-Gaithersburg. Accordingly, greater attention was put on ensuring that ACMD subject matter experts attended relevant Gaithersburg based meetings and workshops and improved teleconferencing capabilities were established at ACMD in Boulder pre-COVID to facilitate high-quality remote meetings. The broader adoption of virtual meeting technologies and practices during the work-from-home period (2020 – 2022) has further reduced the barriers to facile cross-campus communication. With business travel returning to near pre-COVID levels, ACMD is once again sending staff to Gaithersburg as appropriate to ensure the continuation of effective collaborations. Finally, ACMD provides space for five MML HQ staff based at NIST-Boulder. This arrangement further improves ACMD communications with the MML Lab Office and Leadership.

MML Response to Recommendation 10-4:

In line with the Applied Chemicals and Materials Division (ACMD), MML and NIST missions, ACMD maintains a balance of providing (and modernizing) external stakeholder serving services and high-risk high-reward research projects to advance the state-of-the-art metrology in ACMD research domains. At NIST, one funding source for high-risk high-reward research is the internal NIST Innovations in Measurement Sciences (IMS) program. Each year, only two or three new projects are selected for funding NIST-wide and projects generally run for five years. ACMD is part of three active IMS research projects, Measuring Intermolecular Interactions with Electric-Acoustic Spectroscopy (started FY21), Getting From Qubit to Mega-Qubit Quantum Computers with RF Calibrations (started FY21) and Metrology to Enable Hot Qubits (started FY23). The selection process for new FY24 projects is underway and ACMD is leading one and part of three proposals that are still in contention after an initial 50% down-selection, although there is still a long way to go before specific proposals are selected for FY24 project starts. The 2022 CHIPS Act is also creating opportunities for new research at NIST. More specifically, ACMD submitted 13 new project ideas on advancing semiconductor materials metrology to the internal NIST Metrology R&D Program’s recent initial call for ideas. Outside of the NIST IMS and CHIPS programs, ACMD staff are pursuing high-risk research projects in the areas of machine learning for predicting thermophysical properties of mixtures; re-inventing viscosity-liquid-equilibrium (VLE) measurements of mixtures through both NMR and microwave methods; entropy scaling to predict transport (dynamic) properties from equations of state (equilibrium properties); development of human breath metrology (breathomics) for public safety and public health needs; applications of NMR to quantifying CO2 sequestration in building materials; growth and grain-boundary metrology of engineered metallic bi-crystals to advance material property predictions; 3D printing of composite materials for biological and medical applications; and applications of nanoscale topological features in 2D materials to biological, filtering and sensing challenges among others. In summary, ACMD continues to be very actively engaged in advancing the state-of-the-art in science and metrology areas within the NIST mission and our scope of expertise.