An Assessment of the Material Measurement Laboratory at the National Institute of Standards and Technology: Fiscal Year 2023 (2023)
Chapter: 12 Overarching Themes, Key Recommendations, and Chapter-Specific Recommendations from This Assessment
12
Overarching Themes, Key Recommendations, and Chapter-Specific Recommendations from This Assessment
OVERARCHING THEMES AND KEY RECOMMENDATIONS
Adequacy of Facilities and Equipment
As highlighted in the fiscal year (FY) 2020 National Academies of Sciences, Engineering, and Medicine assessment of the Material Measurement Laboratory (MML) (NASEM 2021) and more recently in the 2023 National Academies’ report Technical Assessment of the Capital Facility Needs of the National Institute of Standards and Technology (NASEM 2023), aging facilities and infrastructure present significant handicaps. Notably, the report found that NIST research staff spent from 10 to 40 percent of their time working around facility shortcomings, with the typical reported values being 20 percent. (NASEM 2023) As direct consequence of these limitations, at NIST generally and specific to this report, MML researchers are left to mitigate these limitations through repeating their work or implementing workarounds with limited research budgets.
The MML laboratories in Boulder, Colorado, and Gaithersburg, Maryland, laboratories are negatively affected by poor temperature and humidity control, in addition to inadequate air flow in their laboratories, which are not compatible with state-of-the-art measurements and instrumentation. On the Gaithersburg campus, in 2022, aging infrastructure led to serious flooding of MML research laboratories, resulting in a loss of millions of dollars of equipment and lost time. The facilities are in general inadequate for the performance of modern materials chemistry related processes. Much of the infrastructure, both buildings and equipment, has exceeded its expected life, leading to time spent inefficiently compared to the capabilities of state-of-the-art facilities and equipment in academic and industry laboratories and at metrology institutes around the world. While MML research efforts are world-leading, aging facilities and infrastructure are such that MML is at the cusp of losing its world-leading position and its ability to continue to attract new generations of world-leading researchers.
Key Recommendation 1: The Material Measurement Laboratory (MML) and the National Institute of Standards and Technology Office of Facilities and Property Management should work together to identify and document critical laboratory infrastructure issues. They should collaborate to prioritize items that need to be addressed and create a plan with an associated timeline to address those needs. MML should actively encourage its researchers to document facilities issues and all associated time lost repeating experiments, and the portion of the research budget spent on working around facility issues.
The identified infrastructure issues also represent safety issues: floods can easily lead to sparks that can in turn initiate fires; inadequate exhaust will lead to contaminated laboratory air that can negatively impact research product quality and in the long term can impact the health of research staff. As
a whole, aged infrastructure and equipment can have an impact on the overall attitude of the staff toward safety.
Safety involves an entire organization, including and particularly line management. Managers set the tone and monitor the progress against metrics. They participate in safety inspections. They lead investigations of incidents and near misses. In so doing they create the environment that ultimately grows into a safety culture. Safety is not a grassroots effort, and management needs to do more than establish the objectives; managers at all levels needs to recognize their responsibility and accountability for what happens at the same organizational intensity level as revenue and productivity targets. Participation in the Occupational Safety and Health Administration’s Voluntary Protection Programs might be a useful way to approach safety at MML.1
Key Recommendation 2: Safety is as much a line management responsibility as an individual one. Leaders should recognize that every task can be done safely, and every task has inherent risks. That attitude needs to be transmitted down the line in order for each individual to take responsibility for not only maintaining but also improving the safety of their environment and the environments of those around them. The Material Measurement Laboratory (MML) should maintain statistics of leading indicators of safety risks and facilities dangers that represent the base of the safety pyramid. These statistics should be socialized within the organization in such a way that a safe and safety-conscious environment is created that results in a safety culture that supports continuous improvement. MML should consider engaging with experienced industrial partners who can bring a fresh perspective on the safety risks as a way of providing an unbiased and fresh pair of eyes to the situation.
Adequacy of Scientific and Technical Expertise
MML conducts research at a very high level. It has formal arrangements with renowned institutions such as Brookhaven National Laboratory and the Institute for Bioscience and Biotechnology Research, a joint institute with the University of Maryland, College Park, and the University of Maryland, Baltimore, two campuses of the University of Maryland system. MML has primary responsibility within NIST for producing and curating standard reference materials (SRM) and standard reference data (SRD). In addition to maintaining critical SRM and SRD, MML has developed these products in newer areas standards products based on biology. The balance of maintaining legacy SRM versus repurposing those redirecting limited resources toward producing SRM and SRD in new and emerging areas presents both a challenge and an opportunity. MML notes that areas of new opportunities include the bioeconomy and engineering biology, data science and engineering, and artificial intelligence, and the circular economy.
The robustness of MML’s program against retirements or unforeseen departures could be improved. In some places there needs to be more systematic succession planning to prevent gaps from occurring. Cross-training to provide backup coverage in case the primary cognizant scientist leaves is a further way to protect against disruption. In other instances, retraining staff to take on emerging issues requiring different disciplinary knowledge was lacking.
Pandemic restrictions limited onboarding and other processes that contribute to a cohesive work culture, which has been recognized by the MML leadership team. The lack of cohesion negatively impacts an organization or culture where career mentoring and professional development for the population of term-limited scientific appointees becomes uneven.
Key Recommendation 3: The Material Measurement Laboratory (MML) should work with National Institute of Standards and Technology human resources to develop effective
___________________
1 More information about the Voluntary Protection Programs and how to participate can be found at Occupational Safety and Health Administration, “Voluntary Protection Programs,” https://www.osha.gov/vpp.
processes and procedures for hiring and career development of staff. Processes to ensure timely support to MML in hiring and onboarding of postdocs, term, and permanent employees should be developed and implemented.
Key Recommendation 4: The Material Measurement Laboratory (MML) should require mentorship training for all research staff within the laboratory to ensure quality vertical mentorship and sponsorship in order to increase the impact of the postdoc and early career population both within MML and later in industry, academia, or government positions. MML should increase the number of employees who have formal leadership training to standardize mentoring and onboarding of new personnel. The training should include a formalization of clear mentor and mentee expectations in terms of work, in-person meetings, and supervision. MML should consider a standard message about the processes of transitioning from a postdoc or term appointee to a permanent staff member, and the probability thereof so as to minimize the uncertainty around this process.
Data Management and Data Infrastructure
Data science represents an increasingly important part of MML’s work to meet its mission. Within MML, the Office of Data and Informatics is a dedicated, service-oriented data resource for physical sciences with domain expertise in biological, chemical, and materials sciences, specializing in large and information-rich data sets.
Across MML, data collection, storage, analysis, and movement appear to be handled by multiple, fragmented platforms, and computational data does not appear to be managed in a centralized platform. In addition, not all the experimental measurement systems have a consolidated and automated data infrastructure, which results in a fragmented data infrastructure that, unfortunately, leads to discrete, inefficient, and stovepiped experimental data infrastructure across MML. These observations lead to opportunities to unify all computational systems, data formats, and data transmission protocols into a single, uniform platform for more efficient and effective data curation, storage, processing, transmission, and security management.
Key Recommendation 5: To the extent possible, the Material Measurement Laboratory should launch an initiative to unify all computational systems, data formats, and data transmission protocols into a single, uniform platform for more efficient and effective data curation, storage, processing, transmission, and security management. This data science initiative could be applied to machine learning deployment and expand capabilities in laboratory automation, high-performance computing, and artificial intelligence. Computing infrastructure should, to the extent possible, be updated to meet current laboratory standards.
CHAPTER-SPECIFIC RECOMMENDATIONS
In addition to the major themes and key recommendations above, the panel developed division- and office-specific recommendations in each chapter. Some of these are unique to a given division or office. Others form the basis for the overarching themes of this assessment that led to the key recommendations.
Applied Chemicals and Materials Division
Recommendation 3-1: The Applied Chemicals and Materials Division should develop a process to evaluate the maturity and current and future importance of ongoing projects relative to new opportunities to help decide which programs could be cut in effort or retired completely.
Recommendation 3-2: The Applied Chemicals and Materials Division, and the Material Measurement Laboratory broadly, should have a streamlined and standardized mentoring plan with suitable evaluation of the mentors.
Recommendation 3-3: To maximize outcomes from the use of external funds, the Applied Chemicals and Materials Division should make greater use of technicians for maintaining legacy projects and provide adequate staff time for new initiatives.
Recommendation 3-4: The Applied Chemicals and Materials Division should carry out a detailed market assessment to assess and maximize the return on the long-term investment in generating and maintaining Material Measurement Laboratory’s data-based products.
Biosystems and Biomaterials Division
Recommendation 4-1: Owing to increasing demands on the unit resulting from new bioeconomy initiatives, the Material Measurement Laboratory should prioritize allocations to the Biosystems and Biomaterials Division within budgetary constraints to ensure adequate support and the Biosystems and Biomaterials Division should continue to prioritize its efforts mindful of its limited budget.
Recommendation 4-2: The Material Measurement Laboratory leadership should evaluate space allocations to co-locate the Biosystems and Biomaterials Division in a single building with adequate temperature, humidity, and power control for continuous instrument function.
Recommendation 4-3: The Biosystems and Biomaterials Division leadership should require mentorship training for all laboratory employees and increase the number of employees who have formal leadership training to standardize mentoring and onboarding of new personnel and to make career development opportunities more uniform.
Biomolecular Measurement Division
Recommendation 5-1: The Biomolecular Measurement Division should work with the Material Measurement Laboratory (MML) leadership and the National Institute of Standards and Technology’s headquarters human resources personnel to develop effective procedures for the hiring and career development of staff. Internally, MML should improve their mentorship of postdoctoral fellows to address career development planning.
Recommendation 5-2: The Biomolecular Measurement Division (BMD) should work with Material Measurement Laboratory leadership and the National Institute of Standards and Technology to identify, as soon as possible, laboratory facilities with the level of temperature and humidity control, and space required by BMD to perform its work safely and at a world-class level. A move of BMD to the Institute for Bioscience and Biotechnology Research could be part of this solution.
Recommendation 5-3: The Biomolecular Measurement Division should increase laboratory automation to support critical steps in production of standard products and processes.
Recommendation 5-4: The Biomolecular Measurement Division (BMD) should increase engagement with communities focused on workforce development and forensics analysis, and compile data on how often their resource materials are used. For biomanufacturing workforce development activities, instructional information related to using BMD standards and standard processes could be provided to educational and professional groups providing training for biomanufacturing. Dissemination of information about how often BMD’s published validation studies, research findings, and standard reference materials are used by the stakeholders in forensic activities could increase the appreciation of the value they provide to the legal community.
Chemical Sciences Division
Recommendation 6-1: The Chemical Sciences Division should look for ways to add new scientist- and technician-level staff to continue to service the existing array of standard reference materials (SRMs) and free up staff resources to pursue development of new SRMs. In addition, the division should continue to emphasize staff cross training to ensure functional redundancy. The Chemical Sciences Division should continue to prioritize internal and external collaboration as a solution to maintaining the significant breadth and depth of expertise.
Materials Measurement Science Division
Recommendation 7-1: The Materials Measurement Science Division should consider leveraging its focus area and core competency and capabilities matrix structure to manage the challenge of balancing ongoing base work with the needs of critical emergent initiatives, such as the CHIPS and Science Act of 2022.
Recommendation 7-2: The Materials Measurement Science Division should develop or leverage outside expertise in life-cycle assessment and techno-economic analysis in order to guide and focus its research in the climate mitigation focus area. Likewise, where complex measurement complements complex chemistry (e.g., catalysis, active materials and separations, and sustainable cement), the division should support a more substantial effort in the corresponding chemistries or leverage outside expertise to do so. This could include having a world-class catalyst chemist and metal-organic framework expert working side by side with the division’s materials measurement scientists.
Recommendation 7-3: The Materials Measurement Science Division should consider reallocating budget and staff support to facilitate refurbishment of the division facilities; purchasing of new equipment; increasing the number of technicians, especially in strategic areas or in areas where considerable routine laboratory work is required; and improving the response time and bandwidth of formal Human Resources activities.
Recommendation 7-4: Materials Measurement Science Division management and postdoc advisors should consider a standard message about the chances and processes of transitioning from a postdoc to a permanent staff member to minimize the uncertainty around this process.
Recommendation 7-5: The Materials Measurement Science Division should consider, perhaps in collaboration with other divisions in the Material Measurement Laboratory, establishing a managed data repository, or at least a common data management plan, to aggregate the division’s
data; facilitate sharing across program teams, National Institute of Standards and Technology divisions, and external users; and facilitate data analysis.
Recommendation 7-6: Where appropriate, the Materials Measurement Science Division’s program staff should partner with measurement equipment manufacturers and vendors to facilitate the development and implementation of National Institute of Standards and Technology measurements and standards; this could improve the impact of a range of programs and help identify the quantitative impact of the Division’s work.
Materials Science and Engineering Division
Recommendation 8-1: The Materials Science and Engineering Division should develop more coordinated technical and stakeholder engagement plans for its effort in carbon dioxide capture and the circular economy of plastics and track the long-term impacts of the division’s programs on the nation’s economy and related innovations.
Recommendation 8-2: The Materials Science and Engineering Division should launch an initiative to unify its computational systems, data formats, and data transmission protocols into a single, uniform platform for more efficient and effective data curation, storage, processing, transmission, and security management. This data science initiative could be applied to machine learning deployment and expand capabilities in laboratory automation, high-throughput computation, and artificial intelligence and machine learning methods.
Recommendation 8-3: The Materials Science and Engineering Division should hire more scientists skilled in artificial intelligence, machine learning, and data science to build internal capabilities and capitalize on the growing, diverse sets of data the division has handled.
Recommendation 8-4: The Materials Science and Engineering Division should develop a plan to track the long-term, quantitative impact of its key programs and industrial engagements.
Office of Data and Informatics
Recommendation 9-1: The Material Measurement Laboratory should periodically and frequently engage with industry and academia through activities such as workshops, meetings, conferences, and roundtables to standardize data management frameworks that fully support U.S. science advancement and industry competitiveness via open tools and instruments alike.
Recommendation 9-2: The Material Measurement Laboratory should disseminate broadly the Materials Science and Engineering Division’s protocols for the use of virtual machines to increase the longevity of aging equipment controlled by obsolete computers, effectively empowering industry, academia, and government laboratories to extend the return on capital equipment investments.
Recommendation 9-3: To facilitate the deployment and adoption of state-of-the-art data practices in the Material Measurement Laboratory (MML) and across National Institute of Standards and Technology (NIST), the Office of Data and Informatics and MML should:
- Institute a laboratory-wide policy requiring researchers to use automated data-handling workflows such as those demonstrated in NexusLIMS, and that output data that is easily disseminated through the Public Data Repository with rich meta-data. This will help to ensure that research data generated by MML comply with the FAIR principles of being
- findable, accessible, interoperable, and reusable. All MML researchers should also use the Public Data Repository so that their data are readily available internally and externally.
- Embed laboratory research personnel within the Office of Data and Informatics as a sabbatical or special assignment in order to spread awareness of the data workflows that they have developed and to help to generate new use cases for them under different laboratory conditions.
- Strengthen collaborations with the Information Technology Laboratory to help develop state of the art data practices and disseminate them across NIST.
Recommendation 9-4: The Material Measurement Laboratory should enter more information in the NIST Public Data Repository field to explain why data set metrics are not available where that is the case.
Recommendation 9-5: The Material Measurement Laboratory should strengthen its commitment to the Office of Data and Informatics by establishing clear data governance and data policies. It should establish the National Institute of Standards and Technology (NIST) Public Data Repository as the place to publish all open-source projects and take advantage of its restricted access functionality for any NIST confidential or internal project.
Recommendation 9-6: The Material Measurement Laboratory managers and researchers should adopt the materials science schema developed jointly with the Research Data Alliance.
Recommendation 9-7: The Office of Data and Informatics should collect and make more visible the usage metrics to measure the impact of its repositories. Numbers that report access, download, and reuse (although that is the hardest to measure) would showcase how data and teaching materials are used in and out of government.
Office of Reference Materials
Recommendation 10-1: The Office of Reference Materials should develop and implement a portfolio management process based on the articulated measurement services modernization strategy. The full portfolio of products should be considered critically and monitored on a periodic basis for: (1) each offering’s utility and impact, (2) the resource commitment needed to maintain the product, and (3) effectiveness and performance of translation. Each existing product should have a maintenance and sustaining plan, and a plan to either perpetuate or discontinue products through transition to alternative products or obsolescence. A plan should be developed for every new product proposed or being developed that includes projection of these characteristics, with emphasis on life-cycle analysis, value proposition for stakeholders and value capture for the National Institute of Standards and Technology.
Recommendation 10-2: The Office of Reference Materials (ORM) should benchmark its tasks and the capabilities required to accomplish them against best-of-class capabilities that exist in the private sector. Tasks that are deemed to be better done by outside vendors should be targeted for outsourcing and staffing within ORM should be reallocated accordingly.
Recommendation 10-3: The Office of Reference Materials (ORM) has done an excellent job in managing and distributing product stock, however the marketing effort that leads to sales most often relies on an outside-in approach, in which a potential customer makes a request for a standard. As part of the input for Recommendation 10-1, ORM should initiate an inside-out marketing effort to determine the current value add of each offering in the standard reference
material catalog. This active marketing can increase uptake of languishing products, and at the same time reveal products in the portfolio of limited value proposition. This effort would be expected to increase ORM revenue.
REFERENCES
NASEM (National Academies of Sciences, Engineering, and Medicine). 2021. An Assessment of the Material Measurement Laboratory at the National Institute of Standards and Technology: Fiscal Year 2020. Washington, DC: The National Academies Press. https://doi.org/10.17226/26048.
NASEM. 2023. Technical Assessment of the Capital Facility Needs of the National Institute of Standards and Technology. Washington, DC: The National Academies Press. https://doi.org/10.17226/26684.
