Developing a Strategy to Evaluate the National Climate Assessment (2024)
Chapter: 2 Background of the National Climate Assessment
2
Background of the National Climate Assessment
The history of the U.S. Global Change Research Program (USGCRP) provides important context for the evaluation of the National Climate Assessment (NCA), with several themes becoming apparent. The number and types of audiences for the scientific knowledge of the NCA have increased, affecting the scope of evaluation and creating difficulties in identifying and contacting each audience. Given the Program’s ambition of responding to rapid changes in the scientific understanding of climate change, there are now multiple products to consider when evaluating impact.
This chapter describes the legislative and decision support goals for the NCA, the evolution of the process and context across the five NCAs produced to date, the expanding scope of NCA participants and audiences in the assessment process, and past evaluations of the NCA.
GOALS OF THE NCA
USGCRP’s goals have expanded over time. The legislation defines specific goals in terms of supporting the president and Congress, but the need of multiple audiences for climate information have led to a much broader definition of what the NCA is expected to accomplish.
Legislative Goals
The Global Change Research Act (GCRA) of 19901 established the USGCRP to “improve understanding of global change” through interagency research coordination. The bill was passed unanimously by the Senate, indicating the importance that Congress placed on developing an understanding of global change; in practice, the focus has been on climate change.
The GCRA calls for a research plan that will “combine and interpret data from various sources to produce information readily usable by policymakers attempting to formulate effective strategies for preventing, mitigating, and adapting to the effects of global change” (GCRA § 203, 15 U.S.C. § 2952). It further requires that the Program “consult with actual and potential users of the results of the Program to ensure that such results are useful in developing national and international policy responses to global change” (GCRA § 102, 15 U.S.C. § 2932).
___________________
1 Global Change Research Act of 1990, 15 U.S.C. Chapter 56, Public Law 101-606, 104 Stat. 3096-3104.
Among other requirements, the GCRA mandates the Program to periodically (not less frequently than every 4 years) submit to the president and Congress an assessment regarding the findings of the Program and associated uncertainties, the effects of global change, and current and major long-term trends in global change. The GCRA specifically calls for USGCRP to conduct a scientific assessment that
1) Integrates, evaluates, and interprets the findings of the Program and discusses the scientific uncertainties associated with such findings; 2) analyzes the effects of global change on the natural environment, agriculture, energy production and use, land and water resources, transportation, human health and welfare, human social systems, and biological diversity; and 3) analyzes current trends in global change, both human inducted and natural, and projects major trends for the subsequent 25 to 100 years. (Section 106)
Decision Support Goals
The GCRA also specifies that USGCRP provide policy-relevant information to support decision-making. This goal is described in the most recent NCA (2023), focused on “continuously advancing an inclusive, diverse, and sustained process for assessing and communicating scientific knowledge on the impacts, risks, and vulnerabilities associated with a changing global climate” (USGCRP, 2023e).
The need to consider the effects of climate change for vulnerable populations, disadvantaged communities, and Indigenous peoples/communities has been recognized since the early days of USGCRP. Over time there has also been an important societal increase in comprehension of and emphasis on climate justice, which has been reflected at the federal level (EOP, 2021). One way that the NCA and other efforts have addressed the need for inclusive perspectives is by seeking methods for better integrating multiple kinds of knowledge and knowledge systems (USGCRP, 2023c; Orlove et al., 2023; Whyte, 2017). The first NCA (NAST, 2000) included a chapter on Indigenous peoples that has been maintained through all five iterations of the report. The first report also named the need for audience engagement to understand needs for decision-relevant information, and indicated as one of five research recommendations the need to better understand social systems and the factors that determine vulnerability. In 2014, a USGCRP Social Science Task Force published a white paper and journal article with recommendations on integrating social sciences to facilitate the use of climate information to inform action (Weaver et al., 2014a,b). That led to the formation of an interagency Social Sciences Coordinating Committee that has developed research and recommendations on inclusion of social sciences in the NCA and USGCRP products. The fifth NCA (2023c) introduced the chapter, “Social Systems and Justice,” by saying that social systems are “responsible for the inequitable distribution of both the benefits of energy consumption and the impacts of climate change” (Marino et al., 2023).
In addition to the assessment processes, it is important to note that federal provision of climate services has been developing and expanding over time (see Figure 2-1). The growing sense of urgency around climate solutions (Leiserowitz et al., 2023; Pew Research Center, 2020) has led to a wide range of actors taking steps to address climate change, including subnational governments, civil society, private organizations, individuals, and others (Petzold et al., 2023). The proliferation of solution seekers has also driven an increased demand for climate services (Tart et al., 2020) and greater focus on spatially explicit climate information to serve local needs (NOAA, 2024b). In the past 2 years, the White House Office of Science and Technology Policy (OSTP) convened the Fast Track Action Committee on Climate Services to develop recommendations about how to coordinate climate services across the federal government. The resulting report (NSTC, 2023) included the overarching recommendation for USGCRP to “expand its research coordination role to provide national leadership in coordination and strategic planning of climate services” (p. 2). Such services are defined as “scientifically-based, usable information and products that enhance knowledge and understanding about the impacts of climate change on potential decisions and actions” (OSTP and FEMA, 2021, p. 6). In practice, what constitutes a climate service can take many forms, including information about projected physical, environmental, and social changes and the types of response options that could be adopted. (For an example of the way one federal agency is providing such climate services, see the discussion of the U.S. Department of Agriculture hubs in Chapter 5.) It is important to note that because the NCA and other USGCRP products are themselves climate services, their goals and purpose may change as the federal climate services landscape evolves.
NOTES: The budget crosscut represents the funds identified by USGCRP agencies as their expenditures in support of USGCRP research activities (USGCRP, 2024). Numbers are taken from the federal budget bills enacted by Congress, in millions, demonstrating the expansion of the Program’s work over time. Enacted numbers were not available for 2003–2005.
SOURCE: Generated by the committee based on the budget crosscut reported each year in Our Changing Planet, USGCRP’s annual report to Congress.
PROCESS USED TO CREATE THE NCA
Each of the NCA reports (Karl et al., 2009; Melillo et al., 2014; NAST, 2000; USGCRP, 2018, 2023c) have been led by a team composed of federal and nonfederal scientists and experts from academia, nonprofit organizations, and businesses. The process (e.g., Avery et al., 2023; NAST, 2000) includes opportunities for public comment. Over time, the opportunities for public input have included information-gathering workshops, regional meetings, opportunities to submit information for consideration, and opportunities to submit comments at various stages of report development.
In addition to public review, the reports are reviewed for technical accuracy by USGCRP member agencies. An external panel of experts also provides a detailed review of the draft report (e.g., the President’s Council of Advisors for Science and Technology reviewed the 2000 report [NASEM, 2023]).
Review editors ensure that all comments are addressed adequately. The preface to the archive of public comments and responses to the third-order draft of the 2023 NCA (USGCRP, 2023b) explained,
Each chapter was assigned a Review Editor to evaluate author responses to both the NASEM review and public comments, and the revised chapter drafts themselves, to confirm that the chapter writing teams had given due consideration to all review comments prior to submission for final agency review and clearance. The member agencies of USGCRP sign off on the final report. (p. 1)
The OSTP has sometimes provided a cover letter (e.g., Melillo et al., 2014, p. iii) that becomes part of the report for the delivery to Congress.
The NCA is not policy prescriptive. The GCRA defined it as a scientific assessment, and the first NCA (NAST, 2000) stated “The Assessment’s purpose is to synthesize, evaluate, and report on what we presently know about the potential consequences of climate variability and change for the US” (p. 2).
BROADENING AUDIENCES AND GOALS
Over time, the NCA has increased from a little more than 100 written pages to nearly 2,000 (Table 2-1). Much of that increase is due to expanded science and research (Figure 2-1)—for example, improvements in model performance (IPCC, 2021a) and the narrowing of uncertainty around attribution of events to climate change (IPCC, 2021a; NASEM, 2016). The increased length also reflects heightened public interest in climate solutions (see Box 2-1) (Leiserowitz et al., 2023; Pew Research Center, 2020) and an accompanying desire to provide answers to users. For example, the 2018 (fourth) NCA stated that the key messages in each chapter of the report were intended “to provide answers to specific questions about what is at risk in a particular region or sector and in what way” (USGCRP, 2018, p. 4). This builds on USGCRP’s mandate to provide policy-relevant information to support decision-making. The GCRA, “like most science policies, has a broader purpose—it constructs an aspirational link between science and some form of social progress” (Meyer, 2011, p. 48).
As noted, the GCRA mandates the president and Congress as the NCA’s audience. From the outset, however, USGCRP has recognized the importance of engaging the scientific community and users in preparing the NCA and associated products. The introduction to the initial assessment stated that
climate science is developing rapidly and scientists are increasingly able to project some changes at the regional scale, identifying regional vulnerabilities, and assessing potential regional impacts. . . . This Assessment has begun a national process of research, analysis, and dialogue about the coming changes in climate, their impacts, and what Americans can do to adapt to an uncertain and continuously changing climate. (NAST, 2000, “About this Document”)
All along, these assessments have drawn upon contributions from scientists, as well as
public and private decision-makers, resource and environmental managers, and the general public. The stakeholders from different regions and sectors began the Assessment by articulating their concerns in a series of workshops about climate change impacts in the context of the other major issues they face. In the workshops and subsequent consultations, stakeholders identified priority regional and sector concerns, mobilized specialized expertise, identified potential adaptation options, and provided useful information for decision-makers. The Assessment also involved many scientific experts using advanced methods, models, and results. Further, it has stimulated new scientific research in many areas and identified priority needs for further research. (NAST, 2000, p. 2)
Chapters 4–6 discuss this engagement with a diverse population of participants and audiences and its implication for evaluating the NCA.
TABLE 2-1 Expansion of National Climate Assessments (NCAs) over Time
| NCA Year | Pages | Chapters & Appendixes | |
|---|---|---|---|
| Long Version | Short Version | ||
| 2000 | 522 | 163 | 23 |
| 2009 | N/A | 196 | 25 |
| 2014 | 841 | 148 | 36 |
| 2018a | 1526 | 196 | 34 |
| 2023 | N/Ab | 144 | 42 |
a 2018 details are for “Volume 2”; Volume 1 was a science report of 477 pages released separately.
b The web version of the 2023 report was report of record; printed page counts are not available.
SOURCE: Generated by the committee.
BOX 2-1
Public Interest in Climate Change
Public interest in climate change has increased greatly in recent years, as reflected through the media, dedicated climate communication organizations, and educational resources.
In recent years, the number of news stories that link disaster-causing extreme weather events to climate change drivers has increased (Brimicombe, 2022), both as a result of the increasing frequency and size of major disasters (NCEI, 2024) and because attribution science has matured (IPCC, 2021a; NASEM, 2016). Organizations such as World Weather Attribution, Climate Central, and the American Meteorological Society (Blunden et al., 2023) perform attribution studies of extreme events and summarize their findings in press releases intended for both journalists and the broader public. These and other organizations work to foster literacy among journalists. This may be why explanations of extreme weather in the news media tend to adhere closely to attribution science (Brimicombe, 2022; Clarke et al., 2022).
Public interest is also reflected in the inclusion of climate change in the classroom. More than half of states include climate change in their K–12 science standards (Cho, 2023), although few states provide extensive coverage of the processes involved (Bloom, 2021). Global change is one of nine units in the College Board’s (2021) Advanced Placement Environmental Science course, although no national standards exist for teaching about climate change at the K–12 level (MECCE, 2022). Resources for teaching and learning about climate change are available for educators, parents, and students through governmental agencies (e.g., the Environmental Protection Agency) educational institutions (e.g., Scripps), and nongovernmental organizations such as the Nature Conservancy. At the college level, climate change is taught as a component of courses in many disciplines. A compilation identifies 27 master’s programs in climate science,a typically viewing climate through lenses of energy policy, environmental law, justice, or sustainability. Recently, the State of California established the California Center for Climate Change Education to serve as a training resource for clean energy and green technology jobs at the community college level.
__________________
a See https://www.masterstudies.com/masters-degree/climate-studies.
The role of users of the NCA and related products is reflected in a call to include them in the 2009 (second) assessment (Karl et al., 2009):
A vision for future climate change assessments includes both sustained, extensive stakeholder involvement, and targeted, scientifically rigorous reports that address concerns in a timely fashion. The value of stakeholder involvement includes helping scientists understand what information society wants and needs. In addition, the problem-solving abilities of stakeholders will be essential to designing, initiating, and evaluating mitigation and adaptation strategies and their interactions. The best decisions about these strategies will come when there is widespread understanding of the complex issue of climate change—the science and its many implications for our nation. (p. 158)
Around the same time, the National Research Council released a report that emphasized the need for participation, with a core recommendation summarized as follows:
Public participation improves the quality of federal agencies’ decisions about the environment. Well-managed public involvement also increases the legitimacy of decisions in the eyes of those affected by them, which makes it more likely that the decisions will be implemented effectively. This book recommends that agencies recognize public participation as valuable to their objectives, not just as a formality required by the law.2
Like the first two assessments, the third NCA (NCA3) (Melillo et al., 2014) was developed by a federal advisory committee (FAC) as an institutional mechanism for engaging nonfederal scientists and experts in the
___________________
2 See https://nap.nationalacademies.org/catalog/12434/public-participation-in-environmental-assessment-and-decision-making.
development process. As part of the effort to build on the recommendation for sustained engagement, the FAC was expanded to 60 members and was supported by a team of 300 authors.
Stakeholders involved in the development of the assessment included decision-makers from the public and private sectors, resource and environmental managers, researchers, representatives from businesses and non-governmental organizations, and the general public. More than 70 workshops and listening sessions were held, and thousands of public and expert comments on the draft report provided additional input to the process.” (Melillo et al., 2014, p. iv)
To support the transparency and the usability of the information, the report’s references and data sources were linked in the web version of the report, and a “traceable account” for each key finding explained the evidence and rationale supporting it.
The team for NCA3 sought to develop a process focused on ongoing development of knowledge and incorporation of experience to better support the “ability to understand, predict, assess, and respond to rapid changes in the environment” (Melillo et al., 2014, p. 720). The FAC produced a report (Buizer et al., 2013) with recommendations on increasing engagement in the NCA process to achieve those goals, and NCA3 included a chapter (Hall et al., 2014) on “sustained assessment.”
A group called NCANet was also convened by the FAC and hosted by USGCRP to extend the reach of the NCA process and products for NCA3. NCANet was a “network of networks” (see Chapter 4) with participants from private, nonprofit, and public sectors who were interested in assisting their networks (audiences, communities) in contributing to and using the NCA. Membership was free and open to any group that wished to engage with the NCA in an ongoing way. As the process for NCA3 continued, the number of organizations represented in NCANet grew to more than 200 (see Appendix B). Members saw their networks as both audiences and participants in the NCA, and they participated in development of communications plans and other activities to ensure that their communities would be able to use the information. This network was seen as a key step toward sustained engagement with audiences and development of an assessment process that would produce more timely and useful information. “By cultivating a network of collaborators who connected the NCA to other networks, the NCA3 engagement process laid the groundwork for a sustained assessment” (Cloyd et al., 2016, pp. 39-40). Although NCANet succeeded in creating a network of networks, and the effort was continued through the fourth NCA (NCA4), staff support from USGCRP was subsequently withdrawn, and NCANet did not continue.
Following NCA3, from 2016 to 2017, the Department of Commerce sponsored a 15-member FAC (the Advisory Committee for the Sustained National Climate Assessment) on behalf of USGCRP to provide advice on engagement with audiences and a “sustained assessment” process. When the FAC’s term ended, the members completed their report and recommendations independently (Moss et al., 2019). That paper provides a useful discussion of the case for broad and sustained involvement by audiences and participants in the NCA process and puts forward an ambitious proposal for a civil society consortium to work with USGCRP. Moss et al. (2019) is a useful companion to this committee’s work and describes a concrete way to address the matters discussed in this report.
Around the same time, USGCRP agencies formed the Sustained Assessment Working Group, a federal, interagency group focused on assessment process, strategy, and opportunities. Its stated objective was to “develop a process that includes activities inside and outside the Federal Government, makes efficient use of limited federal resources, and—importantly—is informed by and responsive to evolving user needs” (Avery et al., 2018, p. 1046).
Released in 2018, NCA4 was informed by the prior work and attempted to build on the sustained assessment process. For the first time, the team responsible for development of the report was a federal steering committee composed of agency representatives (as opposed to an FAC with both federal and nonfederal members), and the overall author team grew to more than 500 members. NCA4 also added a number of new chapters in response to public input, including subdividing some regions and adding new topics and contexts. Additionally, NCA4 increased the focus on providing localized information. USGCRP staff and author teams held listening sessions and workshops during the development of the report and engaged audiences through a wide range of presentations.
The fifth NCA, released in 2023, continued to build on the development of the fourth, introducing a chapter on economics, increased foci on social systems and justice, new modes for public engagement (e.g., the Art X Climate initiative), updated figures for improved communication, and further improved the delivery of localized information and accessible data (Avery et al., 2023). The number of authors and contributors again increased, and
author teams and staff conducted outreach and engagement during the development of the report. After the report’s release, USGCRP staff and author teams engaged audiences via new methods, such as chapter-focused webinars, podcasts, and an interactive atlas.
This trajectory of expanding both participants and audiences in the assessment process poses challenges to evaluating the NCA. Perspectives on these challenges and ways to address them are discussed in Chapters 3–6.
PAST EVALUATIONS OF THE NCA
A number of published papers have delved into evaluation of the NCA (e.g., Jacobs et al., 2016; Meyer, 2011; Morgan et al., 2005; Moser, 2005; NRC, 2007; Parson et al., 2003). All of these papers call for continuous learning and evaluation to deliver timely and relevant information to users. Common themes include the need for ongoing user engagement, ongoing assessment processes, and continuous attention to evaluation in order to inform improvement of processes and products. These themes have also been echoed and expanded on in other publications (Buizer et al., 2013, Moss et al., 2019; NASEM, 2017, NRC, 2008, 2013) in ways that are relevant to evaluation of the NCA and related products. These include recommendations for developing an ongoing assessment process and building continuous audience engagement that is thoughtfully scoped to inform development of more useful climate information.
After NCA3 (2014), USGCRP held a workshop on evaluation frameworks, involving experts from across the nation, as a first step toward incorporating ongoing evaluation of the type envisioned for the “sustained assessment” process. The staff produced a report (USGCRP, 2014) summarizing the workshop discussions and recommendations. The workshop focused largely on the process of conducting the assessment, as the process for NCA3 had been reimagined to increase engagement.
The workshop report formed the basis of a post hoc evaluation of the process for NCA3 that was completed by a contracted team of evaluators led by Dantzker Consulting, LLC, in 2016. The evaluation (Dantzker et al., 2016) identified lessons learned and opportunities for improvement, and provided recommendations that can be applied to future NCA planning, development, engagement, and outreach efforts. These efforts were supported by staff from USGCRP and the NCA Technical Support Unit (of the National Oceanic and Atmospheric Administration via North Carolina State University’s North Carolina Institute for Climate Studies). The evaluation addressed questions about the structure and inclusivity of the assessment process, resources and materials available to authors, the writing and review process, communications among the groups involved, the engagement process, distribution of products, perception of products, and the ways in which the assessment informed climate-related decisions.
FINDINGS, CONCLUSIONS, AND RECOMMENDATIONS
Finding 2-1: Since the first NCA, USGCRP has drawn on external contributors, as well as federal agency staff, who have participated in the development and writing of the assessments and related products. The number and variety of these contributors has generally increased with each succeeding NCA.
Finding 2-2: Evaluation has not been a regular aspect of the development process for the NCA or other USGCRP products, nor has it been used to evaluate the outcomes of these products.
Finding 2-3: While the usefulness of a network-of-networks approach to extend the reach of the NCA process and products has been recognized previously, it is not a formal part of USGCRP’s work at present.
Conclusion 2-1: Themes that emerged from the year 2000 to the present are relevant to the evaluation of the NCA and related products; themes include recommendations for an ongoing assessment process, ongoing audience engagement, careful attention to the scope and purpose of audience engagement, and continuous learning and evaluation to deliver timely and relevant information to users.
Conclusion 2-2: USGCRP works within and has responded to a dynamic context that includes dramatic advances in climate science and dramatic increases in the salience of climate change among the public,
eliciting responses in communities, businesses, and subnational governments. Other major changes include a greater focus on spatial variations in the impacts of climate change, as well as increasing attention to the diversity and equity implications of climate change and climate services.
Recommendation 2-1: The U.S. Global Change Research Program, in designing any evaluation of the National Climate Assessment and its other products, should take into account the diversity of participants and audiences with which it seeks to engage.
(The committee further develops Recommendation 2-1 in Chapters 4 and 5.)
Recommendation 2-2: The U.S. Global Change Research Program, in considering evaluation for the National Climate Assessment and its other products, should plan on a strategy for evaluation that allows ongoing learning about how the processes and products are informing decisions, in order to support continuous improvement to its processes and resulting products.
(The committee further discusses Recommendation 2-2 throughout report, including in Chapter 7.)
