6

Implementable Measures for Assessing the Health and Vitality of NASA SMD’s Research Community

A snapshot of the state of the community can be had by examining a broad range of facets associated with the six tenets defined in Chapter 2. However, if change is needed to meet desired goals, the same rigor as that used for evaluating, sorting, and prioritizing the many scientific areas under NASA’s Science Mission Directorate’s (SMD’s) purview may be required in assessing and driving the desired future of the SMD community. Regular and consistent assessments and surveys, accompanied by clear evaluation criteria, in the hands of personnel empowered to act, are needed. The committee focused on both identifying metrics and indicators associated with assessing the current community health and acknowledging that this is a temporal endeavor with its own indicators for progress to plan.

METRICS AND INDICATORS

“Not everything that counts can be counted, and not everything that can be counted counts.”
Attributed to Albert Einstein

In an ideal world, the committee would have identified objective metrics to assess progress and change in the research community. This is a very difficult, if not impossible, task. It is tempting to identify the things that can be counted or measured—for example, number of publications, number of successful grant applications, number of patents, number of people with a variety of demographic characteristics, number of people with managerial responsibility, and more. Furthermore, we are currently measuring things, such as the impact of publications, without fully acknowledging the possible shortcomings of the measures and the unintended consequences of using them.¹

The experience of John Kormendy, a fellow of the National Academy of Sciences,² surrounding his attempt to “calibrate the interpretation of metrics” highlights the issues of unintended consequences from efforts to develop better metrics. His methodology was criticized for perpetuating “academe’s overreliance on simple metrics, to the detriment of scientific discovery.” Critics indicate that women and minorities are relatively under cited in predominantly male, white fields (Flaherty 2021). His work was accepted for publication in the Proceedings of the National Academy of Sciences, and he posted a pre-print on arXiv (Kormendy 2021). He has since taken the pre-print down and suspended publication of his book.

___________________

¹ See, for example, Ginther et al. (2018), Lawrence(2007), Pasterkamp et al. (2007), Urlings et al. (2021), and Wilsdon et al. (2015). ² National Academy of Sciences, “John Kormendy,” http://www.nasonline.org/member-directory/members/20049458.html.

Unlike a metric, which is defined as a standard for measuring or evaluating something, an indicator is a signal that highlights variations, often including a sense of the direction of the variation. Diversity is a concept that can be measured by examining facets of individuals within a system. Even this metric is crude since every individual with the same defining feature does not bring homogenous experiences to the workplace. It is possible, though, to compare the demographic composition of a workforce with that of the general population. Metrics can be used to examine diversity.

Inclusion cannot be measured in the same way diversity is measured. Unlike a metric, which is defined as a standard for measuring or evaluating something, an indicator is a signal that highlights variations, often including a sense of the direction of the variation. There are indicators of an inclusive workplace such as retention and attrition of people with various demographic characteristics. One would expect retention to be similar across demographic characteristics in an inclusive workplace. A focus on inclusion, in addition to diversity, is critical for improving minority experiences in science, technology, engineering, and mathematics (STEM) (Puritty et al. 2017).

The committee has identified six tenets that define a healthy and vital workplace. Ideally, in addition to examining metrics and indicators for each of the six tenets, these metrics and indicators could be combined to give a single, overall indicator of workplace health. Given the span of facets included in the definition, it is not possible to derive a formula or heuristic to combine the measures into one. While not specifically recommended by this committee, SMD may benefit from a single overall indicator. One might imagine a dashboard that includes measures for each of the six tenets and one overall measure. Combining things that are fundamentally different into one meaningful measure is a multi-objective problem. The Analytic Hierarchy Process (AHP) is a tool that allows one person or a group of people to combine multiple metrics and indicators into one meaningful quantitative measure (Saaty 1977).

Metrics, Indicators, and the Six Tenets

This chapter discusses the six tenets in three general categories: demographic metrics and indicators,³ cultural health metrics and indicators that include processes,⁴ and control system metrics and indicators.⁵ Although there is often a focus on the availability and use of the key data on the demographics of the community, data on the culture of an organization provide insight into acceptance—both of members and of novel concepts. Furthermore, the culture of the community data can be used to examine equity in processes and standards of conduct. (Several recent National Academies’ publications note that the cultures of science impact the advancement of science.⁶)

Demographic Data Within NASA

The breadth of the SMD Earth, space, biological, and physical sciences communities extends beyond NASA’s scope, so there are at least two streams of data to consider. There are data to be collected (1) within NASA and (2) from members of the community at large.

The NASA SMD workforce consists of persons employed at NASA headquarters (HQ) as well as the NASA funded civil servants at the NASA field centers and facilities, listed below. However, inclusive of NASA HQ, 93 percent of the center-based civil servants are located predominately at five facilities: the Goddard Space Flight Center, the Ames Research Center, the Langley Research Center, the Johnson Space Center, and the Marshall Space Flight Center (NASA 2021a). In addition, NASA is dependent on two laboratories, the Jet Propulsion Laboratory (JPL), a federally funded research and development center (FFRDC) and the Johns Hopkins University Applied

___________________

³ Captures tenet: Representative Workforce perspective and metrics.

⁴ Captures tenets: Outreach, Acceptance, and Development; Focus from Funding Agencies; Resilience; and Clarity of Science for Efficient Research and Public Support perspectives and metrics.

⁵ Captures tenet: Community Standards of Conduct perspectives and metrics.

⁶ For example, see these reports from the National Academies of Sciences, Engineering, and Medicine: Pathways to Discovery in Astronomy and Astrophysics for the 2020s (2021), Promising Practices for Addressing the Underrepresentation of Women in Science, Engineering, and Medicine (2020), The Science of Effective Mentorship in STEM (2019), Sexual Harassment of Women: Climate, Culture, and Consequences in Academic Sciences, Engineering, and Medicine (2018), and Graduate STEM Education for the 21st Century (2018).

Physics Laboratory (APL), a University Affiliated Research Center (UARC)⁷ to support NASA science missions. The SMD workforce is located across NASA centers and allied facilities:

• Ames Research Center
• Armstrong Flight Research Center
• Glenn Research Center
• Goddard Institute of Space Studies
• Goddard Space Flight Center
• Jet Propulsion Laboratory⁸
• Johnson Space Center
• Katherine Johnson IV & V Facility
• Kennedy Space Center
• Langley Research Center
• Marshall Space Flight Center
• Michoud Assembly Facility
• NASA Engineering and Safety Center
• NASA Headquarters
• NASA Safety Center
• NASA Shared Services Center
• Neil A. Armstrong Test Facility⁹
• Stennis Space Center
• Wallops Flight Facility
• White Sands Test Facility

In February 2021, an internal team commissioned by the associate administrator for SMD delivered a report examining workforce strategies within NASA. Specifically, the report examined the SMD civil servant science workforce within NASA, discussed in Chapter 4 (NASA 2021a). While the approximately 900 NASA civil servants represent a small part of the research population, their actions and decisions set the tone and greatly impact the health and vitality of the broader community.

NASA Human Resources gathers details from the extensive information on its workforce through the NASA Office of Human Capital Management. They have been used to trace changes in the composition of the workforce over time, including the representation of professional degree holders and the positions occupied by women and persons of color. (The trends appear in these two publications: NASA Historical Data Book: Volume IV NASA Resources 1969-1978 and NASA Historical Data Book: Volume VIII: Earth Science and Space Applications, Aeronautics, Technology, and Exploration, Tracking and Dara Acquisition/Space Operations, Facilities and Resources 1989-1998.)

The two large laboratories JPL and APL provided counts of employees and supervisors, but not demographic information for either group. Consequently, this potential source of information on the workforce contains glaring gaps. A second problem exists with the extant information: it does not sufficiently examine changes looming on the horizon due to such emerging phenomena such as “the Great Resignation” and “Global Remote Work.” This limitation appears in a recent analysis on planetary science from the Office of Inspector General (OIG) at NASA. To the OIG, the employment models NASA has set forth pay insufficient attention to retirements, and the models are not sufficiently granular to anticipate the effects that losses of particular skills might generate (NASA 2021d). The research communities that support NASA science missions are constantly gaining and losing people. Losses can occur for many reasons, including retirement. One of the goals of collecting data on these communities is

___________________

⁷ See Johns Hopkins University Applied Physics Laboratory, “About APL,” https://www.jhuapl.edu/About.

⁸ Although JPL is not a NASA center, it is often treated (and referred to) as one. JPL also rivals the largest NASA field centers in number of employees.

⁹ NASA’s Plum Brook Station was renamed the Neil A. Armstrong Test Facility on December 30, 2020.

obviously to determine if losses are occurring for the wrong reasons, such as lack of opportunities, discrimination, or lack of an inclusive climate, so that these causal factors can be addressed. It is normal that a research community will lose senior members who have already contributed much to the space sciences, but NASA does not want to lose junior members driven away before they have the opportunity to contribute. In conjunction with exit interviews, a broader view of successes and opportunities should be apparent.

The data, collected primarily from human resources files, are not sufficient for analyzing the health and vitality of the SMD workforce. First, the information is not obtained from nor aggregated across all the places where civil servants are employed. The limitations appear in the data collected by the 2021 NASA workforce study (NASA 2021a).

Beyond SMD, NASA has an Office of Diversity and Equal Opportunity, which has a Diversity and Data Analytics Division focused on the health of the workforce.

(This plan, required by EEOC under management directive MD 715 is used by agency leadership to ensure equal opportunity and program accountability for NASA programs.) See Box 6-1.

DEMOGRAPHICS DATA ON THE SPACE, EARTH, AND BIOLOGICAL AND PHYSICAL SCIENCES COMMUNITY

Given the broad reach of the scientists contributing to SMD’s research, assessing its demographics is a daunting task. Although extensive material probes metrics on research and the public, diversity in research organizations, as well as the functioning of teams, little of that material draws specifically on experiences within NASA or its research communities.

For the larger community, ideal metrics would capture useful information to support organizational decision-making and strategy.¹⁰ Although the committee developed a list of ideal metrics, the committee recognizes that this is an evolving dialogue. The objective of metrics collection is not building a repository of data, but rather using the information to inform corrective and supportive actions. Because of the dynamic nature of this challenge, social scientists are an ideal contributor to the process. The list of ideal metrics include the following:

• Gender, race, ethnicity, country of birth, and disability status;¹¹
• Type of institution where the person is employed;
• Type of employment;
• Academic background (including NASA experiences as an undergraduate, graduate student, or postdoc); and
• Length of any postdoctoral experience.

These data can be used to assess diversity along many axes, including intersectionalities, and provide indicators for inclusion, retention, potential bias in hiring or promotions, the overall tenure of the workforce by discipline, and more. In collecting data that will be used in assessment of diversity, equity, inclusion, and accessibility (DEIA), great care will need to be taken in the design to ascertain reliable and actionable information. This is not a simple issue; people who are multiracial can identify with a predominant race dependent on context. Social scientists are continually reviewing the best way to collect these data to maximize item response; their input would be useful. The complexities of using terms like Hispanic, Latino, and Asian are also important (Fernandez et al. 2016).¹²

Collection of demographic information must be distinct from proposal submission so that it is not used in the proposal evaluation process (as is currently implemented by SMD). Implementation of the dual anonymous review process, first for the Hubble Space Telescope, and then for other missions and programs, relied on limited subjective determinations (binary gender assignment, seniority, etc.) gleaned from public records post facto. Establishing a process by which this demographic information is collected in an objective, uniform manner across all NASA programs, including those run by external entities such as the Great Observatory science centers, would allow for better statistics, assessment of progress against goals, and identification of areas needing improvement.

“NASA recognizes and supports the benefits of having diverse and inclusive scientific, engineering, and technology communities and fully expects that such values will be reflected in the composition of all proposal teams

___________________

¹⁰ The Council on Quality and Leadership, “12 Reasons Why Data Is Important,” https://www.c-q-l.org/resources/guides/12-reasonswhy-data-is-important.

¹¹ Social scientists continue to examine and refine best practices for asking these questions. Rather than offer the best practices which may be outdated prior to the administration of a survey, we encourage NASA to consult with social scientists prior to survey administration to understand current best practices.

¹² The Federal Committee on Statistical Methods (FCSM) completed a report in 2020 that includes a review of the efforts to date to accurately capture sexual orientation and gender identity (SOGI). The report is available at Federal Committee on Statistical Methods, 2020, “Updates on Terminology of Sexual Orientation and Gender Identity Survey Measures,” https://nces.ed.gov/FCSM/pdf/FCSM_SOGI_Terminology_FY20_Report_FINAL.pdf.

as well as peer review panels, science definition teams, and mission and instrument teams.” This quote is from the 2021 Astrophysics Explorers Mission of Opportunity. DEIA sometimes gets caught in between the lines. In programs where DEIA has been a focus, organizing teams are more innovative and of greater quality.¹³ DEIA is not confined by ethnicity, race, or culture, but includes diversity in skills, educational backgrounds, and disciplines.

The value of diversity beyond ethnicity or race in SMD teams emphasize a diversity in skill mix where teams are often made up of members from aeronautics, technology, and exploration mission directorates with unique expertise that complements the science team.

There are multiple initiatives that NASA, as an agency, has undertaken to collect demographic data on its populations:

• The Office of the Chief Scientist uses an OMB approved form where grant applicants can identify their gender (three options: male, female, other) and other demographic information. The data collected must remain anonymous. The response rate of proposers submitting the voluntary form is more than 80 percent.
• To determine if the award procedures in fact generate teams that match the science and engineering profile of the nation, SMD uses the Office of the Chief Scientist’s survey form to collect demographic data through NSPIRES, the NASA Solicitation and Proposal Integrated Review and Evaluation System. Initiated in 2005, NSPIRES is a computer system that helps the science and technology research community conduct research business with NASA. There are two parts to NSPIRES:

— The public site, which allows access to a list of SMD’s solicitations without becoming a member;

— The member site, which allows response to these solicitations.

    As the system has evolved, NASA asks about gender, race, ethnicity, and disability status for each member of the proposal team through NSPIRES. The submission of information to NSPIRES is strictly voluntary and not included in any evaluation or selection process. Given the voluntary nature of NSPIRES, its demographic information is incomplete, due to the significant numbers of people who select “no response” on questions about their status. Observers have lamented this condition and wondered if revising the phrasing of questions would improve responses. In its current iteration, the data collected from NSPIRES could be used to compare the demographics of people submitting proposals with the general population and with those of people receiving awards. However, the proportion of people who indicate “no response” limits its usefulness in this aspect. In addition, although extensive data exist in the system, NASA or external stakeholders are prohibited access to these data. NASA cannot request demographic data in close out grant reporting. The data are not available to decadal survey teams beyond the incomplete metrics associated with principal investigators (PIs).

___________________

¹³ A lack of diversity in the leadership of an organization hampers innovation, prohibits members of minority groups from being recognized for their contributions, and fails to meet client expectations (Hewlett et al. 2013).

NASA has formal and informal programs that encompass many of these non-NASA communities through EPSCoR¹⁴/MUREP/SpaceGrant¹⁵ and Museum Alliance.¹⁶ Most of these by definition gather demographic data on their members. These data could provide a starting point. Ideally, data collected from the broader community would include factors beyond gender and race/ethnicity. For example, the data could include the type of institution where the person is employed, the type of employment, academic background (including NASA experiences as an undergraduate, graduate student, or postdoc), length of any postdoc experience, number of other faculty, postdocs, and students included in the proposal. Social scientists could help NASA delve into the data beyond bivariate analyses.

In addition to the data collected via the formal programs, NASA could hone the collection of demographic information for all PIs who submit research proposals. While NSPIRES offers those submitting a proposal the opportunity to provide these data, many proposers do not complete the form. In addition, a large proportion of respondents choose “prefer not to answer” for gender.¹⁷ One issue will be de-duplicating the data for PIs who submit multiple proposals and for PIs who are submitting grants for more than one organization (e.g., Goddard and the University of Maryland). One approach NASA could take would be to provide or utilize a recognized unique ID such as an ORCID ID¹⁸ for each PI; that would simplify de-duping. Another complication is the level of support NASA provides to each PI. How the demographic data captures and reports on PIs whose NASA grants comprise a smaller percentage of their funding compared with those for PIs whose NASA grants comprise most or all their funding, or how PIs with multiple grants are counted could be critical.

NASA has sought through various means to characterize and diversify its institutional pool. For example, SMD has made progress in many areas addressing gender diversity across leadership in both management and research scientists. SMD uses a process of inferred gender where first names are used to assume gender. This approach, however, cannot identify gender when people use only their initials. Several reliable tools have been developed to infer name from gender,¹⁹ however, asking people directly, if done appropriately, provides the most reliable data. Dillman (referenced previously) provides an excellent overview of questionnaire development.

___________________

¹⁴ The Established Program to Stimulate Competitive Research (EPSCoR) (https://www.nasa.gov/stem/epscor/about/index.html) is an ongoing NSF-program started in 1978 to facilitate institutional development to ensure federal research and development funds are not overly concentrated in select states.

¹⁵ NASA, 2022, “Minority University Research and Education Project,” NASA STEM Engagement, https://www.nasa.gov/stem/murep/home/index.html.

¹⁶ NASA, “Museum & Informal Education Alliance,” https://informal.jpl.nasa.gov/museum.

¹⁷ L. Barbier, C. Wilson, and A. Kc, 2021, “Committee on Foundations for Assessing Health and Vitality Meeting,” presentation to the committee, August 12, Washington, DC: National Academies of Sciences, Engineering, and Medicine.

¹⁸ ORCID is the Open Research and Contributor ID, a nonproprietary alphanumeric code non-profit that provides free, unique identifiers to uniquely identify those who are engaged in research and scholarship to ensure no ambiguity of person related to publishing of scholarly works, proposals, and other academic public intellectual endeavors. ORCID, “Homepage,” https://www.orcid.org.

¹⁹ These tools include Gender API, genderize.io, and an R routine available at https://cran.r-project.org/web/packages/gender/gender.pdf.

DATA ANALYSIS

As described in several previous decadal surveys and in previous chapters of this report, a healthy and vital research community for SMD benefits from a balance of the science being done at NASA centers and the broader academic and external community. This diversity of thought, career stage, and perspective has resulted, to date, in an innovative, creative, and relevant body of space and Earth sciences.

In a broader context, it is also important that NASA’s research community reflect the diversity of the U.S. population that it inspires and serves. Ideally, this diversity will be reflected in the data that NASA collects, tracks, and interprets. This will not happen overnight as people who identify as women, as African American, as Hispanic, and as disabled are currently underrepresented in the STEM education river. The number of people in the STEM education river who identify as non-binary and as a member of the LGBTQ+ community is very small.²⁰ NSPIRES, the system used to receive, evaluate, and select NASA proposals, is a source of important information to be tracked and assessed. SMD tracks submitted, selected, and success rate of proposals for ROSES research and analysis (R&A) solicitations by institution type: university, NASA center, non-profit organizations, commercial organization, other federal agency, and other. Moreover, the university proposals are broken down into the Carnegie Classification of Very High Research Activity University and less research-intensive institutions.

The committee was also presented with the inferred percentage of women in adaptive optics, solicited science teams by discipline: astrophysics, heliophysics, Earth science, and planetary.^21,22 A notable outlier was the flat percentage of women participating in Earth science since the data were broken out by discipline going back to 2011. No explanation was provided as to this difference relative to the other disciplines. This points to the fact that while the collection of the data is necessary, it is far from sufficient. A resolute and concerted effort also needs to be made to interpret, assess, and address weaknesses identified by the data. These efforts may include programs to increase the representation of people in minoritized groups. Equal representation, however, does not necessarily imply equity.²³ In addition to the examples above, NSPIRES data, even though limited and insufficient to cover the entire research community, should be mined or modified to provide supplementary information on submitted and selected proposals from Minority Serving Institutions, the subset that are Historically Black Colleges and Universities, Hispanic Serving Institutions, Tribal Colleges and Universities, Asian American and Pacific Islander Serving Institutions, Alaska Native and Native Hawaiian Institutions, Native American Serving Non-Tribal Institutions, Predominantly Black Institutions, etc., EPSCoR²⁴ states, community colleges, small teaching colleges, and career stage (assistant, associate, full professor or equivalent). The committee is unable to advise on what are appropriate levels across these various categories; however, it is axiomatic that “you can’t manage what you don’t measure,” and hence the importance of collecting this information and developing baselines upon which to assess future progress. It must also be emphasized yet again that it is not sufficient to just collect the data, it must be analyzed, and mitigation strategies, such as dual anonymous peer review, implemented to ensure progress is made.

___________________

²⁰ See, for example, American Institute of Physics, “TEAM-UP Project,” https://www.aip.org/diversity-initiatives/team-up-task-force; Burke (2007); Burke and Mattis (2019); Out in Science, Technology, Engineering, and Mathematics (oSTEM), “Homepage,” https://www.ostem.org.

²¹ M. New, 2021, “Health and Vitality of SMD’s Scientific Communities,” presentation to the committee, July 19, Washington, DC: National Academies of Sciences, Engineering, and Medicine.

²² No data were presented on the life and microgravity population, likely because the data were assessed prior to the addition of Microgravity and Physical Sciences to SMD.

²³ See, for examples, Allan (2011), NASEM (2020), Payne et al. (2018), and Project Implicit, “Preliminary Information,” https://implicit.harvard.edu/implicit/takeatest.html.

²⁴ Established Program to Stimulate Competitive Research States: The established program to stimulate competitive research (EPSCOR) is designed to fulfill the mandate of the National Science Foundation to promote scientific progress nationwide. See National Science Foundation, “Established Program to Stimulate Competitive Research (EPSCOR),” https://www.nsf.gov/od/oia/programs/epscor.

ALTERNATIVE DATA PARAMETERS AND HOW THEY ARE USEFUL

Traditional demographic data parameters, such as race, ethnicity, gender identity, or age only reveal a portion of a person, and little relative to their true diversity, especially in context of the intellectual space of research. The context of traditional parameters, such as race/ethnicity often does not capture key characteristics of populations—such as the cultural and opportunity differences between a white male living in suburban Boston compared to a white male living in rural Kentucky, let alone the complexities of the full spectrum of recognized races and ethnicities. NASA can consider expanding the scope of its collected demographic parameters to examine the diversity of the current and emerging workforce more fully. NASA will have to decide what data to collect about its workforce as it evolves with respect to remote work and other changes currently happening in the wider workforce, such as the “Great Resignation.”

For the research enterprise, the cultural, intellectual, and experiential diversity of individuals are important contributions to making effective teams (Rock and Grant 2016). Examples of such alternative parameters are found in the American Heart Association,²⁵ Mitroff (1974), Strauss and Connerley (2013), and Tsui and O’Reilly (2017), and include the following:

• Level and institution of highest degree.
• Year highest degree awarded. This is commonly called a person’s “professional age” and often more predictive than biological age as it provides a context for professional experience.
• Prior experiences with NASA or other science agencies. Identifying workforce that was nurtured through internships or postdocs give insight into continuity, and if from other agencies, insights into variances in operational contexts.
• Location at formative points in life. For example, an indicator of formative cultural context is location at the end of secondary school, especially relative to initial resources for development in a STEM professional.
• Undergraduate institution(s). Understanding intellectual pathways add ability to consider outcomes similar to the EPSCOR mission, but also insights into intellectual diversity. The U.S. Shell Oil Research Center in the early 2000s focused heavily on not just their new hires’ terminal school, but their entire pathway. They discovered that there were dominant pathways (undergraduate to graduate program transitions) for a number of disciplinary areas. This yielded predictable, but not diverse teams. They adjusted their investment and recruitment in university programs based on the originating undergraduate institutions, with a focus of fostering a broader engagement, of which one outcome was their support of the Fort Valley State University’s Cooperative Developmental Energy Program.
• An individual’s graduate advisor and their pedigree. In younger fields, distinct “intellectual family trees” have been seen to develop as factions, such as hydrology’s Horton versus Dunne overland flow mechanisms (Bras 1999), so consideration of many parts of pedigree might impact perspectives and background.

When combined with the current information regarding NASA-related employment, these data can be used to examine the distribution of the career stages of workforce members, the clustering of certain disciplines, genders, or racial/ethnic identities, as well as intellectual and opportunity factors within certain fields at NASA.

SMD is moving in the right direction. Changes in leadership have shifted the dynamics of DEIA, and SMD/NASA have solicited input from the science community. Best practice recommendations are most effective when

___________________

²⁵ American Heart Association, “AHA Comments to HHS on Proposed Data Collection Standards,” https://www.heart.org/idc/groups/heartpublic/@wcm/@adv/documents/downloadable/ucm_431557.pdf.

they are informed by the lessons learned of related communities. Trend data will capture forward progress, or opportunities for improvement and means to understand how the distinct components of the SMD community both are similar and distinct (see Box 6-2).

An example of opportunities to engage, attract, and retain key talent has recently been brought to the fore, highlighting another aspect of the health and vitality of NASA’s research community. The recent global pandemic created a change in thinking in the meaning of what is meant by the concept of “workplace.” The shift to remote work has led many companies and organizations to create flexible work schedule arrangements for their employees post-pandemic. In a study assessing drivers and retention strategies for the “Great Resignation,” reported in the Massachusetts Institute of Technology Sloan Management Review, the availability of remote work arrangements was the second highest predictor of successful retention based on employee data. (The presence of lateral career opportunities was the first.) Data from the Microsoft 2021 Work Trend Index: Annual Report (Microsoft 2021) and data from the U.S. Bureau of Labor Statistics Job Turnover and Labor Statistics Survey formed the basis for concluding the “Great Resignation” is real. There is substance for the characterization of a new trend for attrition of American employees, post-pandemic. The all-time records for attrition are ongoing,²⁶ prompting research on predictors of both leaving and retaining employees. Addressing workplace culture, providing opportunities for lateral development and workplace flexibility were identified as emerging retention trends. A key take-away, a thoughtful transition to hybrid work, a compromise between the remote work virtual silos of the pandemic and newly expected work flexibility is a key retention tool (Microsoft 2021; Sull et al. 2022). For NASA to remain an employer of choice among its federal and non-federal workforce, its concept and approach to the workplace may need to change. A culture survey can prove very helpful in defining the flexible work model at NASA post-pandemic to ensure it maintains a mission-focused work environment aligned with NASA’s guiding principles.

To remain competitive in attracting and retaining the qualified and qualifiable, the default workplace will be a hybrid of at-home and in-office/facility work. Employees will expect a flexible workplace post-pandemic.

DATA COLLECTION BEYOND NASA

The professional societies offer an excellent opportunity to characterize community culture. Professional society meetings offer a forum in which researchers can exchange ideas, so they are important to the community. In efforts to create a safe environment for the professional discourse and exchange of ideas, many professional societies are projecting explicit expectations to influence the meeting cultures.²⁷ In addition to the culture of the meetings, NASA could work with the professional societies to conduct a culture survey of the workplace. If a list of members of professional societies was combined as described in Potential Pilot Program 6.2c, the resulting data would be less likely to have multiple responses from one person. NASA could also require institutions that receive a specified level of grant funding to conduct their own culture surveys and provide the findings to NASA. If the agency does so, it will be important to be clear what the agency wants to see from such surveys and why it is interested in the data.

Part of the problem is identifying the full scope of constituent individuals of the broad NASA SMD-supported research enterprise to assess the health and vitality of that community. Researchers come to NASA from a variety of different disciplines and are employed or associated with NASA in several different ways from employee to grant recipient and more. There is no complete list of members of the community. Attempting to compile one is not a one-time effort since individuals are frequently entering and exiting the community and members of the community come from diverse disciplines. (See Boxes 6-1, 6-2, and 6-3.)

___________________

²⁶ U.S. Bureau of Labor Statistics, “Job Openings and Labor Turnover Survey,” https://www.bls.gov/jlt.

²⁷ Many professional societies have formal policies on behavior at their meetings. See, for examples, American Astronomical Society, “Meeting Policies & Conduct,” https://aas.org/meetings/aas239/meeting-policies; American Geophysical Union, “Meetings Code of Conduct,” https://www.agu.org/Plan-for-a-Meeting/AGUMeetings/Meetings-Resources/Meetings-code-of-conduct; Meteoritical Society, “Code of Conduct,” https://www.metsoc2021-chicago.com/code.

In addition, lists of attendees to conferences such as Space Weather Week and the Lunar and Planetary Science Conference help identify members of the relevant communities. These lists provide potential recipients of surveys. Expert, confidential surveys can be used as potential sources of DEIA data to broaden the body of knowledge. Finally, agencies such as NSF and NASA could provide lists of people who have submitted proposals on topics such as life sciences, heliophysics, planetary science, astrophysics, geoscience, and other identified areas. This list of sources is not intended to be comprehensive at this point. The ultimate goal is to identify members of the community, and the community is large and diverse. Establishing ideal performance and associated metrics will rely on multiple external standards and sources.

Thus far, this report has mentioned the possibility of conducting what could be seen as multiple surveys. The

committee encourages NASA to limit the use of surveys while carefully considering which groups should be asked to inform NASA on which topics.

Added sources of information are highly maintained bibliographic databases like GeoRef, which cover segments of the SMD discipline, and SPACELINE, the subset of the PubMed database for biological and physical sciences and human research programs. They can also provide insights into who is actively publishing research on topics and specific programs, both in the peer-reviewed and grey literature.

DATA COLLECTION LIMITATIONS

NASA has a long tradition of excellence in science research, which invariably is driven by engaging the best research workforce available. This totality of the NASA research workforce spans from NASA-employed civil servants to direct contractors to independent grant recipients. This network of people works within an ecosystem of different organizations, each contributing to the stated science missions. But the quality of the resulting research is ultimately NASA’s responsibility, for which they hold the direct managing organizations of the research accountable for their contribution to the missions. This line of responsibility to research could be leveraged toward understanding the dynamics of the entirety of the SMD research community.

To genuinely assess the health and vitality of its research community, the nature of the individuals who constitute the community needs to be understood. Individuals fare better or worse depending on organizational environments. As part of its effort to understand its communities, SMD can characterize the nature of the processes that it puts in place for individuals that constitute regular barriers to advancement. Identifying these barriers is a key step to removing them. Though NASA currently executes various efforts to collect data on the workforce, coordinated data collection instruments to be used across the community would ensure the data collected are consistent in parameters. Not every parameter needs to be included on every instrument but collecting the same data the same way could allow a broader understanding of the community’s composition. The data collected will remain consistent as individuals move between the various entities within the NASA research community—from public to private sector, private to public sector, and between private entities. NASA can consider leveraging their role of ultimate responsibility of the research outcomes as a pathway to also ensure they have ultimate understanding of the people who are contributing to those outcomes, and that all managing organizations should be collecting comparable data and engaging in workforce development policies that meet or exceed SMD’s internal expectations. The mechanisms for this parallelization may well already exist within the grants and contracts in place, and if not, may be a pathway to implementation or through other lines of authority over the science missions themselves.

CHALLENGES IN DATA COLLECTION

As acknowledged in the statement of task (see Appendix A), collection of key data, making it available to appropriate and empowered parties while addressing transparency overall may be problematic. Challenges were identified in a few key areas including legal constraints, accountability for action, and the inherent nature of the research and program selection processes.

NASA is one of several federal agencies prohibited from executing public surveys (paperwork reduction

act)²⁸ without explicit permission from the Office of Management and Budget (OMB). Specifically, the Paperwork Reduction Act (PRA) of 1980, as amended by the Paperwork Reduction Act of 1995 (44 U.S.C. Chapter 35) and OMB regulations of information collections (5 CFR 1320, “Controlling Paperwork Burdens on the Public”) limit NASA’s ability to collect data from employees for “general statistical purposes.” In 2016, NASA received permission to offer a voluntary demographic survey to research proposers. The scope of this survey is limited to researchers and does not include NASA employees. Furthermore, it includes limited gender options and can be updated to fully gain insight into where focused effort should be made.

The White House established the “Equitable Data Working Group” on the first day of the Biden-Harris administration under Executive Order 13985, “Advancing Racial Equity and Support for Underserved Communities Through the Federal Government,”²⁹ as part of the President’s commitment to a whole-of-government approach to advancing equity. As the order notes, “a first step to promoting equity in government action is to gather the data necessary to inform that effort.” Although NASA has an overall representative, it would be helpful if SMD were able to also collaborate with the White House working group to ensure consistent and relevant data are collected across the entire science community.

Data must not be collected unless there is a clear purpose for them. OMB’s “Standards and Guidelines for Statistical Surveys” puts survey planning in Chapter 1; data collection follows in Chapter 2.³⁰ This requirement is not merely a limitation the government places on data collection; developing the research question is the first step in survey research.³¹ Data collection limitations must be considered, as there are limits about what NASA can and cannot ask proposers to submit as part of their requests. The collection of voluntary data; the legality of data collection as well as the dissemination and use of those data must be considered. Resolving issues related to these limitations must be discussed and negotiated with OMB, OPM, and other policy-making entities in the administration in order to eliminate the barriers under which NASA currently operates. An example of a compromise made to enhance access to a more diverse pool of proposals for projects/programs was the implementation of a dual anonymous review process when NASA demonstrated that it was justified because it was based on the observation that the data that were collected were already in the public domain.

An OMB approved demographic survey was created allowing collection of otherwise prohibited demographic data. SMD did not adopt the survey created by the Chief Scientist Office.

___________________

²⁸ The Paperwork Reduction Act (PRA) of 1995 requires that agencies obtain OMB approval before requesting most types of information from the public. “Information collections” include surveys, forms, interviews, and record keeping. The PRA was enacted to minimize the paperwork burden for individuals; small businesses; educational and nonprofit institutions; Federal contractors; State, local and tribal governments; and other persons resulting from the collection of information by or for the federal government. Environmental Protection Agency, 1980, “Summary of the Paperwork Reduction Act” 44 U.S.C. § 3501 et seq., https://www.epa.gov/laws-regulations/summary-paperwork-reductionact. The PRA may be reviewed at https://www.govinfo.gov/content/pkg/PLAW-104publ13/pdf/PLAW-104publ13.pdf.

²⁹ The White House, 2021, Executive Order on Advancing Racial Equity and Support for Underserved Communities Through the Federal Government, https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/27/memorandum-on-restoring-trust-in-governmentthrough-scientific-integrity-and-evidence-based-policymaking.

³⁰ Office of Management and Budget, 2006, Standards and Guidelines for Statistical Surveys, https://www.samhsa.gov/data/sites/default/files/standards_stat_surveys.pdf.

³¹ K. Kelley, B. Clark, V. Brown, and J. Sitzia, 2003, “Good Practice in the Conduct and Reporting of Survey Research,” International Journal for Quality in Healthcare 15(3):261-266, https://doi.org/10.1093/intqhc/mzg031.

SUMMARY OF METRICS AND INDICATORS

Table 6-1 summarizes key metrics and indicators relevant to each of the six tenets. The table includes both data captured within NASA and SMD systems and data available through collaboration with organizations beyond NASA and possible actions. These data are provided to assist in both developing goals and establishing potential actionable opportunities.

NEXT STEPS

Although progress has been significant in the collection of metrics and the intended improvements these data support, much work remains. The next chapter provides reference starting points and possible case studies to inform SMD’s continuing work.

TABLE 6-1 Key Metrics and Indicators

___________________

^a See, for example, C. Williams, 2019, “Simplifying the Expert Elicitation Process,” SpaceWorks commercial, https://www.nasa.gov/sites/default/files/atoms/files/28_simplifying-the-expert-elicitation-process-nasa-cas-symposium-vfinal.pdf and https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.826.1586&rep=rep1&type=pdf.