4

Network Analysis and a Network of Networks

Tracing and assessing the impact of the National Climate Assessment (NCA) and related products requires evaluation approaches that can not only make sense of these products’ immediate usage, but also uncover how their influence ripples outward through indirect connections, as their contents spread throughout networks. The ideas of network analysis are helpful in conceptualizing that transmission of information. For example, by tracing the diffusion of information from the NCA, network analysis can help reveal the various audiences that ultimately utilized the report either directly or indirectly (and perhaps unknowingly through an intermediary) (see Appendix E). Network analysis can also help reveal information diffusion pathways that might support future dissemination efforts. For example, network analysis might help locate potentially underleveraged pathways or cultivate new ones (e.g., emerging opportunities using social media) as new platforms emerge and existing ones evolve over time (see Box 4-1).

Evaluators have used network analysis to understand the impact of interventions that reverberate throughout complex networks of affected organizations (e.g., Smit et al., 2020). By analyzing that network structure, an evaluator can help visualize the complexity of these connections and identify particularly important network connections or points of disconnection or weak connection (Popelier, 2018). This information can help to set priorities for the use of data collection methods, such as surveys and focus groups. In turn, an understanding of the network structure can help the U.S. Global Change Research Program (USGCRP) prioritize audiences and focus its outreach and engagement. In addition, an extensive literature applies network analysis to policy networks, including applications that focus on natural resource management and sustainability (e.g., Henry, 2020, 2023; Henry and Vollan, 2014; Henry et al., 2014, 2021; Weible and Jenkins-Smith, 2016; Weible et al., 2020), as well as studies on climate change science assessment networks specifically (e.g., Corbera et al., 2016; Venturini et al., 2023).

Although USGCRP has recognized the importance of networks for disseminating information from the NCA and its products, as evidenced by the creation of the NCANet more than a decade ago, it has not previously sought to explicitly incorporate the role of networks into evaluation of the NCA through network analysis. Because doing so would be a novel undertaking for USGCRP, the committee provided an overview of network analysis that may be useful as USGCRP considers incorporating this approach into any future evaluation. This chapter introduces ideas from the study of networks and suggests how they may support evaluation of the reach of the NCA and related USGCRP products.

Note that, as presented here, network analysis is a conceptual tool for examining the relationships between the many different groups engaged in climate-related activities, helping to illuminate how USGCRP’s climate information is shared, known, and used. Network analysis is not a replacement for other evaluation methods;

BOX 4-1
Information from the National Climate Assessment Spreads Along Networks

The Great Lakes Integrated Sciences and Assessments (GLISA)—a Climate Adaptation Partnerships program (formerly, the Regional Integrated Sciences and Assessments program) funded by the National Oceanic and Atmospheric Administration—provides an illustration of how information from the National Climate Assessment (NCA) is shared with wider audiences. The figure shows how that information moves from the NCA outward in a partnership with a nongovernmental organization, the Huron River Watershed Council (HRWC). Figure 4-1 depicts only outward transmission of information from the NCA; each of the nodes in the network also receives information from other sources.

GLISA has deliberately enhanced its outreach by funding partnerships that might amplify its ability to connect climate information with users in targeted audiences (Lemos et al., 2014). In this example based on Kirchhoff and colleagues (2015), climate scientists at GLISA work with HRWC to engage with, translate, and tailor climate science for workgroups of officials—such as floodplain managers, water managers, and engineers—representing local governments. In doing so, a boundary organization (e.g., GLISA) takes in available information about climate change (e.g., NCA products), and connects it with constituents in communities throughout this watershed through connections with the HRWC and the local officials with whom the HRWC works. As the information from NCA diffuses through these layers, the immediate connection with the NCA products may be lost, but their influence continues indirectly.

At every layer, relationships may affect the ultimate impact of NCA products, including relationships within and across networks among public officials and constituents. These chains might break, either because of a lack of connections or because a particular link is unreceptive to information about climate change from NCA products (because of climate denial narratives or other factors). The top route in the figure shows a flow of climate information denoted by the arrows from NCA products through HRWC and receptive local officials to constituents. In the bottom route, there is a break in the chain that cuts off this flow of information to whole networks of constituents when local officials are unreceptive. Network analysis may help reveal where such breaks might occur and where opportunities might exist to address them through targeted engagement or circumvent them using alternative networks.

instead, it can play an important complementary role, identifying features of the way information is shared that can be investigated in greater depth by other means. Network analysis can inform and target the use of tools such as interviews, focus groups, and survey questionnaires to collect data about how information is accessed and used, about what works well, about difficulties encountered with respect to making use of the information, and about people’s information needs and how or whether those needs are resolved. It can therefore provide insights that help inform the refinement of the kind of logic model laid out in the preceding chapter. At the same time, interviews, focus groups, and survey questionnaires can be essential for helping identify and measure influence and impacts rippling through indirect connections in these networks (Frank and Xu, 2020; Popelier, 2018).

NETWORK ANALYSIS

Network analysis encompasses a wide array of techniques for studying patterns of connections and how they give rise to outcomes of interest. A distinctive aspect of network analysis is its emphasis on the relationships between things in shaping what happens in the world. Although network analysis was already used prominently in technical work in industries such as communications and transportation, the emergence of the internet and social networking stimulated greater interest in social science applications of network analysis theory and techniques, with striking growth from 1995 to 2010, as online data and analysis software emerged (Brass, 2022). Many detailed and updated introductions are available (e.g., Borgatti, 2024, McLevey et al., 2023); this section has the more modest aim of providing background to inform discussions of how network analysis can support USGCRP product evaluation.

Network analysis features “nodes,” which can represent all kinds of entities (e.g., individuals, organizations, documents), and “links” (or “edges”), which can represent different kinds of relationships between nodes (Box 4-2 provides a summary of some relevant types of connections). Network maps (or graphs) that visualize nodes and the links connecting them are a characteristic feature of network analysis. Originating in the mathematics field of graph theory, network analysis features methods that are generally applicable for understanding phenomena in almost any field, setting, and scale. Network analysis can be highly technical, but it can also be used in mixed-methods research, in conjunction with survey results or qualitative interviews that provide essential ground-truthing of findings or deeper understanding of the phenomena being explored. This flexibility means that network analysis can be used in evaluation in two ways: to gain a road view of how information is spreading and is used in networks, and to improve the efficiency of other methods, including surveys, interviews, and focus groups, so that they obtain more in-depth insight into how specific users interact with information and each other.

The NCA is cited in the communications of some of its important users. Citation networks afford quick understanding of the spread of some important kinds of information. In a citation network study, the nodes are documents and the links are citations made in one document that refer to another document. Academic research and other types of publication—including social network and website postings—are increasingly available in digital form on the internet. Network analysis tools and techniques have emerged to take advantage of these resources (see Kong et al., 2019, for a review). Assessing citation patterns throughout networks of documents and authors citing one another has become particularly prominent in the last couple of decades and can be used for applications

such as exploring the structures of existing fields of study and understanding the impact of a particular piece of information, its diffusion, and patterns associated with its use (Zhao and Strotmann, 2015, p. 2).

Social network analysis techniques have also been widely utilized to understand social media networks (Bazzaz Abkenar et al., 2021; Saganik, 2019) and their impacts on outcomes through influence and information propagation (Chen et al., 2022). Many applications of network analysis rely on far smaller, more targeted data sources. Analyses of meeting attendees or selected documents can be used to understand how information or ideas might spread through networks and identify key nodes connecting otherwise disconnected parts of networks. For example, Braunschweiger (2022) analyzed Swiss adaptation policy development and implementation documents to define a network; they showed how the Swiss Federal Office for the Environment played a unique role as a national authority that connected work across sectors and scales. Interactions at meetings can also provide insight into diversity, equity, inclusion, and justice considerations, such as the marginalization or inclusion of diverse backgrounds and the extent to which efforts to enhance engagement of specific groups are successful (e.g., King et al., 2023; Vasquez et al., 2020). Engagement with key contacts who work with these communities could help identify events to include in a network analysis that could provide important insight into connections that influence the patterns of interactions of underserved audiences. Key contacts might also provide important perspectives about what sources of social media data may best reflect a targeted community’s interactions, as different communities connect using different platforms. Appendix D provides additional details about these methods that may be useful in designing an evaluation of the NCA and related products.

SOME CONCEPTS AND USES OF NETWORK ANALYSIS

Network ideas can contribute valuable perspectives on the transmission of information, and these can be helpful in studying how climate science is transmitted.¹ Identifying the most important or influential nodes in networks is a common goal in network analysis. Centrality measures quantify and rank nodes based on their relative positions within the network. Analysts have developed a wide variety of these measures, to address different ways that the relationships between nodes might affect outcomes of interest; these are discussed further in Appendix D. An analysis of the centrality of federal sources in the transmission of attributions of severe weather events, for example, could provide useful insight about how decision-makers in many sectors perceive the legitimacy of climate information.

Cluster analysis techniques are commonly used to detect clusters (or “communities”) within networks, which are subsets of nodes that are tightly connected with one another but are relatively isolated from other nodes outside of their group. A federal agency might be a cluster, sharing information that could also be of value to other clusters.

Identifying nodes that play a critical bridging role between clusters is associated with some of the most influential concepts in social network analysis related to information exchange. Granovetter (1973) argued that “weak ties” in networks between different groups were critical sources of novel ideas and opportunities within networks. Burt and Celotto (1992) emphasized the pivotal role of individuals who can bridge between “structural holes” in networks and bring together complementary information they might have.

Network analysis can be used to identify these holes and develop interventions to strengthen networks and improve management outcomes (Frank et al., 2023). Network analysis can also help identify and provide support for critical nodes that can bridge structural holes between historically marginalized groups and more well-connected parts of networks (Lopez Hernandez et al., 2022). Social media networks in particular can provide spaces where weak ties between those in historically marginalized communities can make supportive connections with one another that collectively help create and reinforce connections between them and others (Montgomery, 2018).

An example of these ideas at work is the demonstration by Vignola and colleagues (2013) that a regional extension office was a critical node, filling a structural hole for transferring information between larger-scale scientific organizations and more local organizations that were managing agricultural soil loss. USGCRP might

APPLYING NETWORK ANALYSIS TO EVALUATION

To conduct an effective analysis of the reach of the NCA, the evaluator would need to focus the evaluation and identify the limits of the network investigation. That is, USGCRP’s logic model would need to identify key audiences, taking into account their role in the networks where the NCA or a related product are being transmitted or used. Not every member of a network can be interviewed, nor can every chain in the link be investigated. Evaluators might determine the “highest-value” nodes that could help provide insight into how networks spread information and impact information use. Determining the highest value is in part a subjective judgment as to what will best meet the needs of USGCRP and the evaluation team. Highest value might be based on nodes that play key bridging roles or on other criteria described in Appendix D. The evaluation could then engage a selection of nodes of various types to obtain an understanding of the ways that different audiences and users are touching the NCA.

As described above and in Appendix D, citation analysis, weblink analysis, document analysis, or evaluation of the use of NCA and USGCRP products in social media can provide useful estimates of how information in these products spreads and has influence. These techniques might be used to illuminate what types of audiences use these sources, how they might use them differently, and what other sources of information are used concurrently to gain insight into how sources of information supplement one another.

A multilayer analysis in evaluation might be particularly useful for visualizing and assessing the indirect impact of NCA products (Robins et al., 2023). Approaching this as a network of networks multilayer analysis, where documents or authors are differentiated by sector, might be particularly helpful for visualizing and identifying ways in which information spreads between and throughout different sectors. Approaching it as a multiplex multilayer network analysis, in which each layer reflects activities in subsequent years, could potentially highlight the impact that the release of NCA products have, by making comparisons before and after their release. Alternatively, a redundancy analysis could also help quickly visualize the impact that products have, by demonstrating how connections splinter when NCA products are removed from the network.

Findings from a network analysis are likely to suggest to USGCRP and its evaluator the ways in which the logic model should be reconsidered. Viewing the potential influence of NCA products as a flow between and within different layers of networks, as illustrated in Box 4-1, emphasizes that this influence is shaped by the perceptions and decisions of individuals within these networks. Mapping networks might help define mediators of the NCA and other users that could support the definition of audiences in the logic model or even reveal unexpected pathways to pursue that weren’t initially incorporated in the model. This can provide critical insights about audiences, partners, and connections between them for addressing the “Who With” aspect of the logic model. However, additional efforts might be needed to explore “How They Feel,” “What They Gain,” and “What They Do,” which also influence the NCA’s impact as it travels through nodes on these pathways.

Complementary approaches can shed light on users’ perceptions and experiences that can help inform understanding of these aspects of the logic model, their connections between one another, and—ultimately—the impact of the NCA. For example, the mental model construct is an approach for exploring how conceptions that people hold about the world within which they are acting impact their responses to information, interactions with each other, and actions (Jones et al., 2011). Evaluators have used different qualitative and quantitative approaches to explore these conceptions and their impacts. Hoffman and colleagues (2014) used a network analysis to identify key connections between concepts in farmers’ definitions of sustainable agriculture that shaped their participation in extension programs and adoption of sustainable practices. Research on policy networks has also emphasized that the shape of networks can provide insight into understanding “What They Do,” as network locations can influence diffusion of information, learning, and actions (e.g., Henry, 2016). At the same time, while network structure can impact collaboration (e.g., Henry, 2023), participants’ perceptions can also impact the choices they make about the relationships that produce the network structure (Feiock et al., 2012; Lee et al., 2012).

It is important to recognize that network methods present persistent challenges related to causality. Networks are inherently complex systems in which participants impact one another in overlapping and often indirect ways. Network analysts use approaches such as longitudinal data and sensitivity analysis to help interpret the impact of networks on outcomes. However, explanations about the effect of networks should also be judged based on how well they hold up relative to other causal explanations for observations, rather than on whether or not they offer definitive explanations of cause and effect (Frank and Xu, 2020). To that end, network analysis may also provide

insights for how to evaluate and discuss causality related to complex dynamics and outcomes that USGCRP is interested in, consistent with the discussion of contribution analysis in Chapter 3.

Network findings can also help to guide targeted efforts to explore subsets of the network of networks in greater depth. Box 4-3 provides a list of potential candidate audiences for investigation through narrower assessments. A detailed discussion of potential audiences and their prioritization is provided in Chapter 5.

General Approach

This section discusses a general approach to collecting and analyzing data on the role of networks in extending the reach of the NCA. As described earlier in this chapter, some of the appropriate techniques include citation analysis (examining who cites the NCA, and how that fits in a broader pattern of citations), study of weblinks (e.g., who provides links to the NCA, along with the pattern of other links that they provide, then expanding to a larger network of links), and social media (e.g., those who follow the NCA and reference it with hashtags). Similarly, although this would greatly expand the scope of the analysis, one might look at hashtags for prominent client science deniers, examining how people using such connections are connected to those referencing the NCA. With information on the structure of the network from automated tools, an evaluation can use interviews and survey questionnaires to learn how network organizers perceive their networks, whom they seek to reach, and what tools they use to reach their networks. However, when incorporating an analysis of the role of networks into an evaluation, careful consideration must be given to the resources required for this, since collection and analysis of network data can be time-intensive and therefore needs to be included in the assessment budget.

Evaluation Questions

Network analysis provides approaches that can help answer the evaluation questions proposed in Chapter 3. It can shed light on how information from the NCA and USGCRP products was shared. Evaluators could engage persons in significant nodes to provide insights as to whether they shared USGCRP information or products with their network and whether they used the NCA in planning or preparing products to share with their network.

Indirect Users

An important purpose of mapping a network’s reach is to understand the indirect audiences that might be missed in other data collection or analysis activities. A network of networks analysis can provide an understanding of the indirect users and highlight gaps in audiences. In turn, this information will allow USGCRP to prioritize audiences—either those who are engaged or those who are missed—and develop specialized products and targeted outreach to serve them.

Underserved Audiences

The network perspective also provides a useful means of finding some of the audiences that are missed as climate information spreads. In mapping the network carrying the NCA and related products, an evaluation identifies actors in the nodes of the network. These persons understand the audiences in their network and can provide helpful insights on which of those audiences is not using climate science information in their decision-making, either because they do not find it useful or because they are not aware of sources such as the NCA and derivative products that use climate science information. People in these nodes are thus a significant audience because they understand where and how the content of the NCA is not getting through. In turn, the audiences that are identified by the key nodes can be studied so that USGCRP can gain an understanding of how information transmission dissipates as the NCA moves outward through networks.

Informants in nodes of the network are likely to have insight into their own reach and the gaps that they wish to fill. They could be asked questions including Who are their audiences? Which of their audiences could make use of information the NCA or derivative products in their decision-making? Why are they not doing so—are they unaware of the usefulness of climate science information? Do they believe that such information is useful? Informants in nodes can also be asked Whom do they wish they could better engage? What kind of participation do they have from vulnerable communities? Are there nodes within their network that engage directly with vulnerable and disadvantaged communities? Nodes’ understanding of their networks can suggest changes in the NCA logic model by identifying gaps and stimulating further evaluation of efforts to reach groups that are not being reached but are important to USGCRP.

It is important to keep in mind, however, that the network approach does not identify all audiences that could use the information in the NCA in their decision-making. Audiences that are not known to the nodes within the NCA network will not be found by looking through that network. In practice, as noted above, an evaluation cannot reach all the nodes in the network used by USGCRP, which further narrows the audiences of nonusers that can be found. Given the challenges of finding those who are not connected, directly or indirectly, to a network, even the limited insights that can be gleaned by including important node actors in the evaluation may be helpful. Evaluators may follow up with nonusers who are identified initially to pursue further insight into others that they may be aware of who are currently not connected to the NCA’s networks.

Prioritizing Audiences

As described in Chapter 5, USGCRP will need to apply a set of criteria to prioritize NCA audiences for inclusion in the evaluation. Network analysis may reveal valuable information about particular organizations’ roles in transmitting knowledge from the NCA, and it is important to take this into account in setting priorities. If the goal is to understand all users of the NCA and related products, the evaluation can use a network analysis to assemble an audience that includes nodes of various types. For example, an evaluation might include one or more of each type of node discussed in Box 4-3. Representation would also be useful across nodes that are focused on climate or view it as secondary, that have large and small audiences, that serve different sectors or regions, that participate in the NCA process, or that use the NCA as a resource.

An illustrative example is a network analysis of the NCA’s contributions to climate services. USGCRP has a direct role in coordinating federal climate services efforts (see Chapter 2) and is connected to the federal programs providing climate services in communities across the nation,² which are themselves nodes in the network