Developing a Strategy to Evaluate the National Climate Assessment (2024)
Chapter: 6 Methodologies and Applications to Particular Audiences
6
Methodologies and Applications to Particular Audiences
Chapter 5 discusses how different audiences offer different benefits to an evaluation and different levels of feasibility for investigation. It offers criteria for prioritizing audiences and considers how the differences among audiences require different evaluation methodologies for specific audiences. This chapter discusses evaluation methods, with a particular focus on methods for examining specific audiences, building on the discussions in Chapter 4 on the ability of network analysis to identify links connecting audiences and the properties of those links as a connected system.
To develop a methodological framework for an evaluation, it is critical to align the following: (1) what the evaluation seeks to learn (as articulated in the logic model and overarching evaluation questions, using the illustrative examples from Chapter 3), (2) from or about whom you need to hear to answer those questions (based on prioritization using the criteria from Chapter 5), and (3) the attributes of different evaluation methods. This chapter begins by providing a very brief overview of some of those methods and offering considerations for how different methods may be applied in combination to answer overarching evaluation questions. This is followed by considerations for tailoring those methodological approaches to priority audiences. Given the large number and diversity of audiences, no single approach will be consistently effective and feasible. This chapter discusses why customized strategies are needed and gives illustrative examples of how to develop customized strategies.
OVERVIEW OF POTENTIAL METHODOLOGIES
This section distinguishes between collecting primary data (e.g., surveys, interviews, and focus groups conducted specifically for the evaluation) and secondary data (e.g., data collected for other purposes that may be used to help answer overarching evaluation questions). After providing a brief overview of selected methods and their advantages and disadvantages, the chapter discusses considerations specific to evaluations conducted by the U.S. Global Change Research Program (USGCRP). Case studies, which may combine primary and secondary data to explore a particular instance in depth, are then highlighted as particularly relevant to USGCRP evaluations.
While this section describes each method separately, it is important to note that they are often deployed in combination. For example, network analysis (as described in Chapter 4) might identify subjects for case studies. Evaluators may conduct a survey first and then follow up with more in-depth interviews to gather nuanced information about survey findings—or they might conduct interviews first to refine what questions to ask in a survey.
(More information about sequencing of different evaluation methods can be found in Chapters 7 and 8.) Using multiple methods in combination is also consistent with the iterative approach of building support for contribution analysis, as it provides multiple opportunities to compile evidence to build, test, and revise the understanding of the link between cause and effect (Mayne, 2011).
This section concludes by drawing connections between the overarching evaluation questions from Chapter 3 and the methods described here.
Collecting Primary Data
Chapter 5 articulates a series of criteria that can be used to prioritize which participants should be incorporated into an evaluation to gain insights about their use of the National Climate Assessment (NCA) and other USGCRP products.
Overview of Collecting Primary Data
Mechanisms for collecting information directly from those prioritized audiences (primary data) include surveys, interviews, focus groups, and observations. Paradis and colleagues (2016) describe the “ideal” use of each of those methods:
- Surveys: to capture perceptions, attitudes, beliefs, or knowledge of a defined sample of individuals. Surveys are often also used as a data collection tool for large populations.
- In-depth interviews: to capture accounts of, perceptions of, or stories about situations or phenomena from individuals.
- Focus groups: to understand phenomena when “the sum of a group of people’s experiences may offer more than a single individual’s experiences” as well as to document how participants react to what they hear from others.
- Observations: to document what individuals actually do, rather than to ask them about their perceptions or recollections about what they do.
In addition to identifying which of those purposes best addresses the evaluation questions, there are practical considerations with respect to the different modes of data collection. Table 6-1, which draws from the Centers for Disease Control and Prevention (CDC, 2011) and Taherdoost (2021), highlights some of the advantages and disadvantages of each of these four methods. The table is not meant to be exhaustive, nor is the list of data collection methods. Instead, this table is meant to illustrate the types of trade-offs that will be encountered in determining an appropriate combination of methods. Evaluators can provide more in-depth guidance about the benefits and limitations of each method and suggest other possible data collection methods.
Considerations for USGCRP Evaluations
When determining which data collection methods to apply, it is important to begin by grounding the decisions in what evaluators are seeking to learn, as captured in the overarching evaluation questions and their associated subquestions (Chapter 3). This will help evaluators identify which method(s) are most likely to yield answers to the evaluation questions.
For example, surveys may be best suited to contribute to understanding the level of awareness about the NCA from a large number of audience (or potential audience) members, which relates to Evaluation Question 1 (“To what extent are priority audiences aware of NCA products and what are the most effective ways to increase awareness?”). On the other hand, to understand why priority audiences take action, interviews may provide the opportunity to hear the story behind a decision to act, which relates to Evaluation Question 4 (“How did the attributes of the products and process contribute to how users feel, what they gain (e.g., cognition), and what they do (e.g., behaviors)? What about the products and process could be changed to make them more effective?”).
TABLE 6-1 Selected Advantages and Disadvantages of Data Collection Methods
| Selected Advantages | Selected Disadvantages |
|---|---|
| Method: Surveys | |
|
|
| Method: Interviews (individual/in-depth) | |
|
|
| Method: Focus Groups | |
|
|
| Method: Observations | |
|
|
Given their interactive nature, focus groups may be particularly valuable for informing how the NCA is viewed by audiences within the context of the broader climate information ecosystems, per Evaluation Question 5 (“How do the contextual factors described in the logic model influence how audiences feel, what they gain [e.g., cognition], what they do [e.g., behaviors], and how they mediate the use of the NCA?”). Having a group of participants discuss their varied sources of information about climate in a focus group might produce a more comprehensive list of alternative sources and therefore foster greater discussion about how NCA compares with and complements those other sources.
Evaluation Questions 4 and 5 can provide insight into how historically marginalized communities engage with the NCA and its development process. During its open meetings, the committee heard that simply inviting individuals to join part of the NCA development process may be insufficient for truly engaging them in the discussion. Observations might yield valuable insights about how diverse participants engage in NCA deliberations and development and what opportunities might exist to foster truly inclusive discussions. Based on the observations, selected individuals might be interviewed to understand more about whether their viewpoints were properly represented, and what barriers or encouragements they may have encountered.
While evaluations that seek to answer multiple complex overarching evaluation questions will likely seek a mix of different methods, that specific mix will depend both on the questions to be answered and on practical considerations, including some of the concerns raised above.
Considering the feasibility of fielding surveys with adequate and representative responses, it may be appropriate for the evaluators to seek to add questions to existing surveys of the prioritized group, rather than fielding a separate survey. This will also help to reduce survey fatigue by limiting the number of surveys being sent to the same group of individuals. This method depends on finding a survey that is targeted to the desired population and on the ability to ask questions of sufficient depth in a survey that may be designed for a different purpose.
For in-depth interviews and focus groups, which are seeking to answer questions more about how and why, rather than how many, the need to have a statistically representative sample may be less significant than gaining insights on the mechanisms behind understanding, decisions, or actions. Rather than using a statistical approach to sampling, qualitative data collection often uses “purposeful sampling,” which seeks to identify “information-rich cases for the most effective use of limited resources” (Palinkas et al., 2015, p. 534). This concept is further explored in the discussion of case studies below.
Document Review and Analysis of Secondary Data
In addition to collecting new data, compiling and analyzing existing data can yield important insights in evaluation (Vartanian, 2010).
Overview of Using Secondary Data
Secondary data are those that were collected for a purpose other than the evaluation at hand. They can include program reports, data routinely collected by organizations for other purposes (e.g., attendance at webinars), media analysis, and information in the scientific literature, among others. Some of the advantages of using secondary data include avoiding duplication and the potential for saving time and money. Secondary data can also provide additional context for primary data and therefore a broader understanding of the question at hand. As such, secondary data analysis is often used to complement primary data collection to address overarching evaluation questions more holistically. However, because secondary data are collected for other purposes, the validity of the analysis will depend on the quality of those data sources. What is more, the concepts and measurements used in secondary data may not exactly match those that are the focus of the evaluation (McCaston, 2005; NCVO, n.d.). Secondary data may already exist in a readily usable form (as in a general survey that included questions about climate change) or may require additional compilation or processing to prepare the data for analysis.
Considerations for USGCRP Evaluations
The logic model can help inform hypotheses about the types of documents that are most likely to demonstrate NCA’s contribution to decision-making and help determine what types of databases might be most valuable to search. For example, an evaluation team may theorize that use of the NCA or related products in the development of policy documents is a critical pathway for informing decision-making. If so, evaluators would specifically tailor their search approach to capture policy documents.
Textual review of existing documents can be used to complement primary data collection. For example, an evaluator may review the websites of a handful of U.S. conservation and environmental organizations to determine which ones appear to be using USGCRP products, to identify organizations for initial interviews. Depending on the final set of overarching evaluation questions, the evaluators may seek to reach out to a selection of both those organizations that appear to use USGCRP products (to learn more about their use) and those who appear not to use it (to learn more about underserved populations and their reasons for not using those products). Conversely, if an evaluation were to begin by administering a survey to public health officials, the evaluation team might use those responses to identify examples of health departments that are actively engaged in climate change work and use that information to focus their review of websites or documentation on those organizations that have indicated engagement in this area.
Some forms of secondary data analysis may be particularly beneficial to support continuous improvement. Web analytics of USGCRP pages could not only help identify which parts of the NCA and related products are most frequently being viewed, but also provide insights on user experience with the website (Palomino et al., 2021). Similarly, web scraping might be used to gather information about where the NCA is cited or mentioned on the
web in general or within particular communities of practice (Mitchell, 2018). This might inform decisions on areas of focus and web and interaction designs for future NCA reports. Similarly, there is a growing body of tools and frameworks for tracking how social media and news media are spreading content (Almasy and Thompson, 2013; AMEC, 2024). An evaluator might help USGCRP establish targets and systems for ongoing tracking in these areas, to support continuous improvement of USGCRP outreach efforts. Given the emphasis on diversity, equity, and inclusion, it would be helpful to collect demographic information, where possible, from individuals who register for USGCRP events or sign up for the newsletter.
Case Studies
The U.S. Government Accountability Office (GAO, 1987) defines a case study as “a method for learning about a complex instance, based on a comprehensive understanding of that instance obtained by extensive description and analysis of that instance taken as a whole and in its context” (p. 9). Put another way, case studies provide in-depth information about a particular organization or scenario, often drawing from collection and analysis of data from a variety of sources (Balbach, 1999). For example, evaluators may analyze program documents, conduct a survey, and engage in multiple interviews all about one organization. This methodology may be especially relevant for USGCRP, as it allows an evaluator to gain an in-depth understanding of the mechanisms by which something occurs. As Yin (2009) explains, “[t]he more that your questions seek to explain some present circumstance (e.g., ‘how’ or ‘why’ some social phenomenon works), the more that the case study method will be relevant” (p. 4).
While some researchers emphasize that a focus on a small number of cases limits the ability to create findings that can be used to make statistically significant statements about the impact of a program (one understanding of the word generalizability), the case study method is particularly useful for illuminating the theory behind the intervention, and that can help build understanding of how or why a program may be effective (Tsang, 2014; USAID, 2013). As such, case studies may be especially effective in drawing lessons learned and information that can be applied to improving the intervention—or, in this case, supporting continuous improvement of USGCRP products and processes.
Multiple types of case studies (USAID, 2013) might be applicable to USGCRP. A critical instance case study describes why a specific case is unusual. In network analysis, this type of analysis may be helpful to explore why and how a particular node appears to be so influential. For example, it might be appropriate to conduct a case study of a nonprofit organization that appears to be especially effective in translating information from USGCRP into formats used by its members or constituents. Determining what makes that organization successful could yield information relevant to other organizations. This aligns with a positive deviance approach, which uses statistical analysis to identify cases that perform better than their peers with similar characteristics. Focusing case studies on those outliers can reveal insights about what makes some organizations particularly effective (Klaiman et al., 2016).
An explanatory case study investigates the effects of a particular program and seeks to make causal inferences about the success (or lack thereof) of the program. For example, USGCRP could conduct such a case study of a climate-relevant policy that was enacted by a state or local governmental entity to understand how USGCRP contributed to that outcome and what worked (or did not work) well. Such findings can be applied to continuously improve USGCRP processes and products.
Comparative case study designs look at multiple cases to identify similarities and differences. This type of approach could be used, for instance, to see how different federal climate programs have used USGCRP products and could yield important insights on the network of networks approach.
Connecting Methods to USGCRP Evaluations
The range of methods selected in an evaluation always needs to be based on what the evaluation is seeking to learn. As such, Table 6-2 provides methodological considerations for each of the illustrative overarching evaluation questions listed in Chapter 3.
The left column provides examples of how those illustrative overarching questions, which broadly describe what can be learned from the evaluation comprehensively, can be translated into questions that are specific enough to be included in a survey or interview/focus group guide. This list is not exhaustive, but it illustrates how the overarching questions, as well as specific elements from the logic model, are translated into more granular questions.
TABLE 6-2 Data Collection Methods and Considerations for Overarching Evaluation Questions
| Primary Data Collection: Examples of Questions for Surveys, Interviews, Focus Groups | Considerations for Analysis of Both Primary and Secondary Data |
|---|---|
| Evaluation Question 1: To what extent are priority audiences aware of NCA products and what are the most effective ways to increase awareness? How, if at all, did involvement in the development process contribute to general awareness and use of the report? | |
|
Secondary data analysis:
|
| Evaluation Question 2: How, and to what extent, did NCA products address information needs among priority audiences (i.e., what did they gain cognitively in terms of knowledge, skills, attitudes, capacities, etc.)? Evaluation Question 3: How, and to what extent, did NCA products address decision needs among priority audiences (i.e., what did they do as a result of using the products)? |
|
|
Secondary data analysis:
|
| Primary Data Collection: Examples of Questions for Surveys, Interviews, Focus Groups | Considerations for Analysis of Both Primary and Secondary Data |
|---|---|
| Evaluation Question 4: How did the attributes of the products and process contribute to how users feel, what they gain (e.g., cognition), and what they do (e.g., behaviors)? What about the products and process could be changed to make them more effective? | |
|
Secondary data analysis
|
| Evaluation Question 5: How do the contextual factors described in the logic model influence how audiences feel, what they gain (e.g., cognition), what they do (e.g., behaviors), and how they mediate the use of the NCA? | |
|
Secondary data analysis
|
| Evaluation Question 6: How does the network of networks factor into use? | |
|
Secondary data analysis
|
NOTES: NCA = National Climate Assessment; USGCRP = U.S. Global Change Research Program.
Also note that questions in surveys are more likely to have close-ended response options than those used in interview and focus group guides. To simplify matters, examples of questions that might be used for all three types of protocols are included below. Questions that are particularly suited for interviews or focus groups are included in italics. However, the evaluation team would refine these questions and, as described in Chapter 3, may engage in cognitive interviewing or other processes to fine-tune the questions and increase the likelihood that they will yield information in a manner that will benefit the evaluation.
The right column provides some considerations about analysis of data—either analysis of data collected through the evaluation itself (primary data) or analysis of secondary sources. It is also designed to provide examples of potential methods that are appropriate for the overarching evaluation questions, rather than a complete list of potential analytic tools.
ILLUSTRATIVE APPLICATIONS TO PARTICULAR AUDIENCES
This section considers five examples that illustrate how USGCRP can use a logic model, combined with the overarching goals of an evaluation and evaluation questions, to plan for data collection and other information-gathering activities from different types of audiences. These examples were selected to demonstrate how a diversity of users can be taken into account in designing an evaluation, as well as how the choice of evaluation methods can be tailored to not only the specific audience of interest but also the specific goal of the evaluation as it relates to use of the NCA by each audience. The examples are (1) federal agencies or programs, such as the U.S. Department of Agriculture (USDA) climate hubs, which collectively constitute a major audience for the NCA and also function as important nodes in a network of users; (2) community groups and nongovernmental organizations (NGOs), which can also provide nodes to networks that touch audiences that might be difficult or impossible for USGCRP to reach directly; (3) public health professionals, who serve a wide range of roles and interact with individuals and communities on many levels (from very local to national); (4) mass and social media, which form a widespread and diverse network focused on information sharing; and (5) K–12 educators, who reach audiences of important future decision-makers but are difficult to access directly. The purpose of the discussion here is not to imply that these five groups should be the priority audiences for inclusion in an evaluation of the NCA—that prioritization needs to be determined by USGCRP following criteria such as those outlined in Chapter 5. Rather, it is to illustrate the diversity of pathways through which audiences might make use of the NCA and its products, and how the Program needs to identify those pathways and associated key evaluation goals in order to choose appropriate methods and design plans for data collection.
Federal Agencies
Federal agencies both contribute to the NCA and provide important immediate nodes in the network of networks through which information in the NCA is shared. Many federal programs provide climate services and make NCA information and products available to regional, state, local, tribal, and other collaborators. Each program has an institutional commitment to a set of users and user pathways: each is designed to pursue and to facilitate uses of climate information, including that provided via the NCA. Examples of important federal programs that serve as both users and potential nodes in sharing information about or from the NCA include:
- National Oceanic and Atmospheric Administration’s (NOAA’s) Climate Adaptation Partnerships/Regional Integrated Sciences and Assessments program
- NOAA’s National Centers for Environmental Information Regional Climate Center Program
- USDA’s climate hubs
- U.S. Geological Survey’s climate adaptation science centers
- U.S. Fish and Wildlife Service national landscape conservation cooperatives (LCCs)
- U.S. Department of Energy’s Grid Resilience Technical Assistance Consortium1
- U.S. Department of Health and Human Services’s (HHS’s) Office of Climate Change and Health Equity (OCCHE)
___________________
1 See https://www.energy.gov/gdo/grid-resilience-technical-assistance-consortium.
These listed programs particularly have a role in translating and adapting climate science for use in local applications; there also may be value in looking at other operations within the federal agencies and how they make use of the NCA.
USDA’s Climate Hubs
USDA’s (n.d.-c) climate hubs illustrate how the framework developed in Chapters 3 and 4 could be applied to federal agencies’ involvement in the NCA. The 11 hubs anchor collaborations “within their regions, which include scientists, practitioners, and local decision-makers, to identify regional vulnerabilities, scientific needs and data, and methods for mitigating impacts and adapting to changing conditions” (U.S. Climate Resilience Toolkit, n.d., p. 1). (See Appendix F for more detail regarding the operations and services provided by the climate hubs.) Each hub is tasked with addressing the unique climate challenges and opportunities of its region, while also working with other hubs to develop and share information nationwide (USDA, 2021). The USDA hub employees combine the job of researcher with the job of agricultural extension agent, forming a key link between the NCA and the agricultural, ranching, and forest management communities. They provide periodic regional assessments of risk and vulnerability to production sectors and rural economies, building on material provided through the NCA. USDA hub employees also serve as contributors to the NCA.
Additionally, USDA climate hubs collaborate with other federal agencies and work together with state, local, and tribal governments, as well as with universities. Together, the federal network and their collaborators at a regional level constitute a network of networks addressing “actionable scientific information that enables local decision-makers to work collaboratively, across landscapes, to implement adaptation strategies that reduce risk and build resilience” (U.S. Climate Resilience Toolkit, n.d., p. 2). The hubs form central nodes in a network that includes the following (USDA, 2021):
- Internal partners: USDA research agencies and program agencies.
- External collaborators: Other federal agencies, universities and extension, states, tribes, local governments, nongovernmental organizations, international organizations, and other technology transfer providers, including but not limited to natural resource managers and service providers, and regional boundary organizations.
- USDA customers: Farmers, ranchers, and forest managers, and the people and communities that depend on them. (p. 10)
Considerations for USGCRP Evaluations
To evaluate the use of NCA by federal agencies, using the climate hubs as an example, a first step is for USGCRP to clearly identify the goals of the evaluation and what they hope to learn from it (see Chapter 3), because this will inform not only the specific questions to ask but also the approach or methodology to use for data collection. For example, if the goal is to learn the extent to which there is familiarity with or use of NCA products within the hubs, then USGCRP could conduct a survey across all hubs, asking specific questions such as those described in the left column (data collection questions) of Table 6-2, above. Since the hubs are a well-defined, easily identifiable, and relatively small audience (11 hubs), a census (sent to all hubs rather than a representative sample) is feasible. The questions could be structured to collect quantitative data using, for example, prepopulated lists of potential responses rather than open-ended questions. The survey results could then be collated, summarized numerically, and reported.
Alternatively, if USGCRP wants to understand how the hubs are using the NCA and its products and how they might use them more effectively, then it might want to select a diverse subset of the hubs and conduct in-depth case studies of each, including interviews of key personnel, focus groups, etc. In this case, the interview questions could be more open-ended, asking, for example, where hub staff get their climate-related information, what they do to transform it, and what their dissemination strategies are. This would yield qualitative data that provide a more in-depth (but potentially less easily numerically summarized) picture of NCA use by the hubs.
Given the important role that the hubs play in sharing climate-related information and collaborating with other groups, the evaluation could map out the key networks through which the hubs operate and serve as a node.
This mapping accounts for transmission that is both vertical (to and from senior leadership) and horizontal (to and from federal and other collaborators, as well as outward to customers). After identifying these networks, information could then be gathered on the extent and ways in which the hubs use and share information about the NCA through these networks. Interviews, focus groups, and case studies are likely to be more effective than surveys in generating an in-depth understanding of the ways in which the hubs facilitate use of NCA-generated information by members in their network, particularly those members who are not easily identifiable by the evaluator and may not even be aware of the source of the information received through the hubs.
Finally, a practical consideration is that evaluation instruments can be used with federal employees without clearance procedures by the Office of Management and Budget (OMB), which are required by the Paperwork Reduction Act. While this requirement can be addressed, the evaluator and USGCRP should consider the appropriate times and places when focusing on only federal employees may offer tactical advantages.
Quick Summary: USDA Climate Hubs
Role in network of networks: information providers—they have internal partners within USDA; external partners including federal agencies, universities, states, tribes, and local governments; and USDA customers (e.g., farmers, ranchers, and forest managers)
Uses of NCA (actual or potential): translate and deliver NCA information (coupled with other information sources)
Membership: 11 hubs, well defined
Accessibility for evaluation: high—membership lists exist, OMB clearance not needed
Community Groups and NGOs
Many community groups value information about climate change. As used here, the term community is construed broadly. It can be place-based and unique to a particular group of people that lives there (Fitzgerald, 2018) or apply to individuals who share a common religion, core identity, or objective. Community boundaries can be perceptual (such as an ethnic identity) or actual (such as living in a specific neighborhood) and are frequently defined by inclusion and exclusion criteria (Fitzgerald, 2018). Clearly, different communities will have different informational and other needs (as reflected in the illustrative logic model in Chapter 3).
Communities may be direct users of the NCA and its products, but may also be indirect users through, for example, materials generated by NGOs. Many NGOs (e.g., 501(c)(3) or (4) organizations, public charities) represent or support community members by advocating for policy changes, offering services, raising awareness, and/or mobilizing resources to spearhead localized change. These entities can use the localized data from NCA products to inform their communities about the potential impacts of climate change, possibly via derivative products that make use of the NCA without the target audience necessarily being aware of the reliance on NCA information.
Union of Concerned Scientists
The Union of Concerned Scientists (UCS) is an example of an NGO that acts as an intermediary between the NCA and the general public. UCS is a nonprofit organization created to put science into action. Climate change is a priority topic for the group, and the NCA is one of their major information sources, along with National Weather Service data and alerts concerning heat, fire, floods, and storms. As an intermediary between the NCA and the public, UCS tailors and translates NCA information to fit local situations. For instance, UCS creates maps as an outreach tool to show how local communities are affected by climate change and how corresponding inequities arise, both directly through effects of the weather and indirectly such as through impacts on the energy grid. Through these efforts, UCS can provide localized assessments that go well beyond what is available directly in the NCA.
Considerations for USGCRP Evaluations
Given the broad definition of relevant communities, identifying the full set of community groups that might make use of the NCA for inclusion in an evaluation would be infeasible. Rather, USGCRP can engage a subset of groups in an evaluation (through, e.g., in-depth interviews or case studies) to appraise the success of the NCA process in understanding their needs and co-developing products tailored to meeting those needs. Under this approach, USGCRP could identify communities of high priority based on its logic model and what it wants to learn from the evaluation. The selection of communities could include, for example, communities where the effects of climate change are significant and where information of the type provided in the NCA might be particularly useful. It could also consider diversity, equity, and inclusion goals in selecting communities, particularly those that might face barriers to awareness, access, or use of the NCA and its products.
Alternatively, USGCRP could focus on evaluating indirect use by focusing on NGOs that make use of the NCA in preparing derivative materials tailored to specific community needs. This is likely to be a more identifiable audience. If USGCRP wants to see how widespread the awareness and use of the NCA are among NGOs and the extent to which they share information about the NCA, it could conduct a survey of major environmental nonprofit organizations such as UCS, The Nature Conservancy, Sierra Club, Environmental Defense Fund, and the Natural Resources Defense Council. It could also look at NGO websites to see the extent to which those organizations are utilizing knowledge from the NCA. If, on the other hand, USGCRP wants to learn about how NGO engagement with local communities impacts local decision-making, a case study of an organization such as UCS that works closely with local communities and provides localized information based on the NCA could be conducted to gain insight into how NCA information is translated for use at the local level and how future assessments and related products could facilitate this bridging function.
Quick Summary: Community Groups and NGOs
Role in network of networks: varied—they may create and disseminate information and/or be actively involved in addressing climate change, often at the local level; community groups may be users only
Uses of NCA (actual or potential): translate and deliver NCA information (and possibly other data sources); use of NCA information for planning and decision-making
Membership: an estimated 1.5 million NGOs working in the United States; the number involved in climate change activities is unknown; the number of community groups is unknown
Accessibility for evaluation: one might examine one or a few selected NGOs and/or community groups
Public Health Professionals
Public health is a policy arena that encompasses a wide range of occupations, united by concern for the health of human populations. This section discusses how the NCA is responding to public health concerns and illustrates methods that could be brought to bear in evaluating the outcomes of the NCA on this heterogeneous audience. Moreover, this example can be extrapolated to the many similar professional networks that intersect with climate impacts across a range of disciplines.
Public Health Professionals and Climate Science
The relevance of USGCRP to public health is reflected in the many parts of the fifth NCA that specifically address issues salient to public health research and practice, including Chapter 15 (Human Health), Chapter 31 (Adaptation), Chapter F3 (COVID-19 and Climate Change), and mentions of health threats in other topically and regionally focused
chapters. Public health professionals are integral to the identification of climate impacts on public health (Frumkin et al., 2008; Romanello et al., 2023), estimation of health co-benefits of climate action (Roca-Barceló et al., 2024), advocacy for climate change mitigation (Kreslake et al. 2018), and evaluation of climate adaptation efforts (Joseph et al., 2023; Turek-Hankins et al., 2021). They hold roles such as science communicators, researchers, governmental staff, and employees of NGOs and private-sector companies; they work across multiple sectors and can be found in academic, state government, nonprofit, and research institutions. In these roles, they might reciprocally contribute evidence that informs USGCRP and refer to the NCA and other products to contextualize their work (USGCRP, 2016a). In addition, federal scientists work in agencies with a public health mandate (e.g., U.S. Centers for Disease Control and Prevention, HHS’s OCCHE) (USGCRP, n.d.), which may be major users of USGCRP products and may produce secondary products that influence public health practitioners in state, tribal, territorial, and local agencies.
Public health practitioners working in state, tribal, territorial, and local agencies also play critical roles in tracking population health outcomes and supporting public outreach and engagement on climate adaptation measure implementation (Albright et al., 2020; Errett et al., 2022; Frumkin et al., 2008; Kreslake et al., 2018; Marinucci et al., 2014; Romanello et al., 2023; Rudolph and Gould, 2015; Sheehan et al., 2017). These professionals may be consumers of the NCA and related products as a credible source of information on which to base outreach efforts. Other resources, including those compiled by HHS’s OCCHE, are also oriented toward community public health practitioners and might be tapped in an evaluation of the NCA and related products.
Considerations for USGCRP Evaluations
Public health professionals constitute a diverse community of practice, and the degree of direct and indirect use of NCA products and information will vary with the individual’s role. For example, the perceptions surrounding NCA products and their effects on knowledge and capabilities and on decisions and practice may differ according to the type of public health worker (e.g., county agency staff, state research scientist, academic researcher). Evaluation of the use of the NCA and its related products by this audience would require that the evaluation questions and methods be tailored to a subgroup that has been prioritized using the criteria in Chapter 5 and based on the logic model (see Chapter 3) and the overall goals of the evaluation.
For example, if USGCRP wants to understand use of the NCA by local public health agency professionals, they might conduct a survey that includes questions from Table 6-2 about norms (e.g., “Is this what others like me are using?”), efficacy beliefs (e.g., “Does this help me to do my job?”), and control beliefs (e.g., “Is this easy to access?”). Those norms and beliefs might influence the probability that the individual would use a USGCRP product to “learn about climate data,” “understand solutions and what can be done by myself and others,” or “access and use climate data.” As an alternative to conducting its own survey, USGCRP could seek to add questions to surveys already being conducted of those professionals such as the National Association of County and City Health Officials’ Forces of Change survey2 or the Public Health Workforce Information and Needs Survey.3
In addition, since the public health community has multiple sources of information on climate and health, including the CDC, fact sheets produced by the American Public Health Association (APHA), and information provided by OCCHE, it would also benefit an evaluator to engage with some of these agencies’ staff to learn about whether and how they use NCA products to inform materials for local public health practitioners. It is also important for the evaluator to recognize that there may be differences in perceptions about norms and accessibility between USGCRP original products and translational secondary products (e.g., fact sheets).
If USGCRP’s goal is instead to understand how CDC researchers use the NCA in their research, a different set of questions would be needed. For example, questions may focus on whether the data provided by USGCRP (e.g., in the NCA Atlas) are granular enough and formatted in such a way to allow for easy analysis. USGCRP could also conduct citation analyses of CDC publications—both scientific publications (e.g., the Morbidity and Mortality Weekly Report4) and those designed for a broader audience, such as newsletters—to see the extent to which they reference the NCA.
___________________
2 See https://www.naccho.org/resources/lhd-research/forces-of-change.
Quick Summary: Public Health Professionals
Role in network of networks: varied—they may be more likely to be users of NCA information than creators or disseminators of climate change information
Uses of NCA (actual or potential): translate and deliver NCA information (and possibly other data sources); use NCA information for planning and decision-making
Membership: categories may include epidemiologists, public health nurses, health educators, environmental health specialists, health administrators/managers, biostatisticians, and health policy analysts; they are highly dispersed—APHA estimates there are a minimum of 500,000 in the public health workforce (Perlino, 2006)
Accessibility for evaluation: complete membership lists not available; might start by looking at a single public health organization or to look at a more narrowly prescribed subset of public health professionals (e.g., local health department officials)
Media
The mainstream media is a network through which information from the NCA and its products is distributed. This broadly includes entities that generate and distribute content via cable, print, broadcast, and digital television and those that use contemporary media platforms such as social media, short-form media, and the worldwide web. Decision-makers and the American public follow the news, and the news that the media choose to cover or not cover often sets the social agenda (Simons, 2021). Some news outlets report on a broad range of topics, while others focus on specific topics. For example, Climate Central is a nonprofit news organization that both conducts research and reports on climate science and the impacts of climate change.
Media as a Network of Networks
Furthermore, media operate through a network of networks, as they continue to influence public life. A news article could report an NCA finding, which would then be discussed and reported on online platforms (McNamara, 2009). For example, a television reporter or journalist could turn to the NCA to obtain information on climate change or to use data to create localized footage for their specific television or print market, which could in turn be picked up by other media. Similarly, media could pass on institutional knowledge and best practices regarding the value of the NCA to their peers, further affecting the dissemination of NCA products.
The news media face daunting obstacles in spreading awareness and accurate information about topics of interest to the public. These include the closure of regional newspapers (Gorman, 2011); a highly politicized atmosphere (Bykov, 2021) in which people turn to the media to reinforce their preexisting views; and the proliferation of misinformation, disinformation, and malinformation (Hunter, 2023).5 Climate misinformation and its spread through the media, particularly social media, is an issue of particular concern. For example, in a review article, researchers noted that there appear to be organized networks supporting climate misinformation and that these networks are highly organized in the production and dissemination of misinformation (Treen et al., 2020; Turrentine, 2022). An NCA evaluation could shed light on what role, if any, NCA information plays in these misinformation networks and how these practices could be countered. Moreover, scholars have also demonstrated that strategies for countering disinformation encompass, among other things, educating the public about disinformation and “innoculating” them by disseminating examples of disinformation (Roozenbeek et al., 2020). Evaluators therefore need to be aware of the possibility of misinformation and disinformation
___________________
5 Information may be problematic because it is false but with no intention of causing harm (misinformation), false with the intention to create harm (disinformation), or based on fact but used out of context to mislead, harm, or manipulate.
when evaluating NCA use and consider measures to assess whether or not users have made accommodations to address this problem.
As an adjunct to the mainstream media, use of social media has been argued to serve as a way to overcome the power of media elites and make media distribution, curation, and creation more accessible and democratic (Anstead, 2015). The climate youth movement and its sister movements have relied heavily on social media to spread their messages and rally people to their cause (Fisher, 2012). The spread and use of social media, particularly by younger audiences, represents an opportunity to enhance awareness and use of NCA products—however, social media can also contribute to misinformation, disinformation, and malinformation (Van der Linder et al., 2017).
Considerations for USGCRP Evaluations
To learn how extensively the media contribute to sharing information about the NCA and its products, USGCRP could use media analysis, including, for example, a scraping of newspaper stories or social media posts for mention of NCA. As part of a process of ongoing evaluation, USGCRP could also subscribe to media-tracking software such as TVEyes Media Monitoring for ongoing use. Such data can provide measures of the extent and nature of the sharing of information about the NCA throughout the media network, and the types of individuals and organizations that are reporting on the NCA and its products. A focus on explicit mentions would not include all the ways in which NCA influences the media, however, as USGCRP products are also used indirectly (with or without attribution) in media products related to climate change.
An evaluation of how the media are using the NCA could include surveys, interviews, or focus groups of journalists, as well as search tools such as LexisNexis. Science journalists represent one audience within the broader media audience that may rely on NCA products to help them carry out their duties. Identifiable audiences include the Society of Environmental Journalists and Internews’ Earth Journalism Network. Other groups, such as the Society of Professional Journalists or the National Association of Hispanic Journalists could be included if USGCRP wants to learn about coverage of climate change and climate science.
Alternatively, to understand more fully how the media are using (or not using) the NCA, USGCRP could do an in-depth case study of use by an organization such as Climate Central. Such a case study could draw on interviews with staff (scientists and journalists), as well as website and citation analysis.
Quick Summary: Media
Role in network of networks: their primary role is to disseminate information
Uses of NCA (actual or potential): deliver information from the NCA and other sources
Membership: 3,000 newsroom outlets, plus blogs, podcasts, talk radio, and other digital content
Accessibility for evaluation: could be examined through media search tools or a case study
K–12 Educators
One audience of significance is the population of the future, which is likely to be exposed to increasingly severe impacts attributable to a changing climate. A portion of that audience is today’s students, whose experience of the NCA can be evaluated via their teachers. The challenge of including educators in an evaluation of the NCA illustrates some of the difficulty of understanding how climate science reaches, and fails to reach, the general public.
Increasingly, K–12 educators are teaching their students about climate change, including how the climate is changing and how those changes impact both their own lives and the lives of others, now and into the future. Some educators also emphasize resilience strategies, to support students who may be impacted by extreme weather events (Dubois and Krasny, 2016).
The NCA and its associated products can support educators and students in two ways. The first is by providing information to those educators so that they can learn about climate change and use that knowledge in their educational mission. For this, educators might access the NCA directly or might learn about it and the information in it through another audience, as part of a network. The second is by providing resources that can be used in the classroom to teach students about climate change in a way that will be effective and engaging. In all areas, including climate change, educators face the task of communicating complex information to students in ways that are tailored to students’ needs (Kirk et al., 2014; Monroe et al., 2019), which scholars have demonstrated is crucial if the intention is for the audience to understand and act upon the message (Cummings, 2017; Nisbet, 2009). Moreover, as busy professionals (Walker, 2023) for whom climate science is not a primary mission topic, K–12 educators are likely looking for easy-to-share, easy-to-understand, and trustworthy information they can use in their classrooms.
According to a review article analyzing 959 studies on climate education interventions, successful environmental education had two things in common: (1) content was personalized and made specifically relevant to the student, and (2) teachers used active, engaged learning while working hands-on with students (Monroe et al., 2019). Educators seek trustworthy information to introduce into the classroom and often rely on established sources, including listings on the internet (Puttick and Talks, 2021).
Educators and the NCA
Some NCA-related tools that could be particularly useful to educators in the classroom include the U.S. Climate Resilience Toolkit6 and the NCA Interactive Atlas.7 For example, students may be motivated to learn more about the communities they care about by seeing how farmers are adapting their growing methods to cultivate strawberries in Florida’s changing climate or how the urban heat island effect is influencing Chicago.
Researchers have found that teachers often rely on four types of information sources when it comes to educating students on climate change: the internet, government sources, the media, and professional development courses, according to a scoping review of nearly 600 papers (Puttick and Talks, 2021). Sites that aggregate listings for teachers, such as commonsense.org, may be worth investigating, since educators use them to identify trustworthy sources for teaching about climate. This website, for example, lists National Geographic Education and NASA Global Climate Change—Vital Signs of the Planet as top sources, but does not reference the NCA.
Considerations for USGCRP Evaluations
USGCRP could evaluate the use of the NCA by K–12 educators through a network of networks approach, first looking at whether there are particular educator nodes that the NCA is or is not collaborating with to disseminate awareness of NCA or that are being used to shape derivative products. Possible key educator nodes include professional organizations for teachers and K–12 leaders, Ted-Ed, Scholastic Education, PBS Learning Media, and Open Educational Resource Commons. Within these nodes awareness of specific tools should be evaluated (see Behl et al., 2015) using focus groups, surveys, or one-on-one interviews. Given the importance of education for providing underserved populations with climate information, an evaluation might also identify target subpopulations based on diversity, equity, inclusion, and justice considerations, to determine, for example, whether NCA products or tools are being used in ways that improve resilience of economically marginalized groups.
USGCRP could also collect information on the use of the NCA by reviewing curricula and programming for references to the NCA or NCA products. Sources for review could include resources listed on commonsense.org or the climate change resources provided by the National Science Teachers Association.
___________________
6 See https://toolkit.climate.gov/content/us-climate-resilience-toolkit.
Quick Summary: K–12 educators
Role in network of networks: their primary role is to disseminate information
Uses of NCA (actual or potential): probably through intermediary sources
Membership: 3.8 million full- and part-time public school teachers
Accessibility for evaluation: no list exists; could be surveyed through two-stage sample (schools first, and teachers within schools); could review curricula or look at educator nodes
CONCLUSIONS AND RECOMMENDATIONS
Conclusion 6-1: A variety of methods is needed in an evaluation of the NCA; some of these collect and analyze primary data while others are used to analyze existing data. Each has advantages and disadvantages, and they vary in the extent to which they address different types of questions. Surveys are well suited to gathering some information from a larger number of individuals and focus groups, while in-depth interviews, case studies, and observations can provide more nuanced understanding of how climate science information is used and how it is disseminated through networks.
Conclusion 6-2: Audiences make use of the NCA and its products through diverse pathways. Identifying those pathways and associated key evaluation goals is needed in order to choose appropriate methods and to design plans for data collection.
Conclusion 6-3: Audiences will differ in their characteristics, including how easily they can be identified and contacted for data collection. This will impact the feasibility of using different methods to collect primary data and availability of existing (secondary) data.
Recommendation 6-1: The U.S. Global Change Research Program should design its evaluation and data collection plan so that the methods used for priority audiences can answer the overall evaluation questions identified in the logic model. The methods and approach chosen should be tailored to the audience and the evaluation question being investigated.
