Developing a Strategy to Evaluate the National Climate Assessment (2024)
Chapter: Appendix D: Network Analysis: Additional Measures and Strategies
Appendix D
Network Analysis: Additional Measures and Strategies
Network analysis features concepts and techniques that could potentially be of value to those involved with the evaluation of the use of NCA products. In the process of developing the approaches used across many different contexts and applications, however, network analysts have created a great deal of technical terminology and methodology specific to their perspectives on networks. These terms and methods are not widely familiar, and that can limit recognition of opportunities to use network analysis within an evaluation, complicating communication with network analysis experts. Chapter 4 provides an overview of how network analysis could support exploration of a network of networks associated with the use of NCA products. This appendix provides additional context about network analysis measures and strategies that are anticipated to be relevant for addressing the type of evaluation discussed in this report. It is meant to aid those involved in the evaluation who do not have backgrounds in network analysis with making sense of how network analysis might support their efforts and how to communicate with network analysts.
A basic distinction that network analysts commonly make is between different types of networks. The most common types are egocentric, monopartite, and bipartite networks. An egocentric network focuses on a particular node at the center with connections extending from it, such as asking an individual who they collaborate with the most and then mapping the network based on those individuals and their links to other nodes branching off from them. Such an approach might be used with USGCRP at the network’s center to develop a map of the network of networks as perceived by the program.
Most network maps do not start with a central node at the outset and instead focus on the larger system. Monopartite networks have only one kind of node, such as a network where peer-reviewed publications are the nodes and links are made between them based on which papers cite one another. It is also common to use bipartite networks, which have two kinds of nodes. If the citation network was approached as a bipartite network, it could include both the peer-reviewed publications and the people who authored them. In that network map, authors would be linked with particular papers and then the papers would be linked with one another. Event data are also commonly mapped as a bipartite network where participants are linked with particular events. Monopartite and bipartite networks are also commonly referred to as one-mode and two-mode networks, respectively.
As mentioned in Chapter 4, there are different ways network analysts assess the centrality of nodes based on how they want to understand how the location of nodes might affect outcomes of interest. Understanding which node has the most total connections can be an indicator of importance, and degree describes the number of links a particular node has with the other nodes in a network. However, a node might have many direct ties with a particular
set of links in a network, but then be relatively isolated from many other parts of the network. Closeness addresses this potential isolation relative to all the other nodes in the network by calculating the average number of links that need to be traversed to get from any particular node to the other nodes in the network. Another consideration is the influence a node ultimately has on the network, which might be determined by examining the influence of the other directly linked nodes. Eigenvector centrality ranks nodes based on their connection with highly ranked nodes. Google’s PageRank algorithm, which allowed its original search engine to innovatively identify and promote higher-quality webpage results, is an adaptation of eigenvector centrality (Brin and Page, 1998). Sometimes the importance of nodes in a network is not about the number of connections, but about how critical the node’s position is for connecting parts of the network that are isolated from one another. Betweenness describes the number of times a particular node is a part of the shortest route between two other nodes in the network. Identifying such key connections between parts of a network that are otherwise disconnected can be a powerful application of network analysis. The importance of those who bridge between structural holes in networks, described in Chapter 4, is associated with the recognition of the importance of nodes with high betweenness.
Chapter 4 highlights citation analysis as an established application of network analysis that can support an evaluation, but there are different ways that it can be approached depending on what is trying to be understood. Different types of relationships within these citation networks may be of interest, with common distinctions made between patterns of direct citation (one thing cites another), co-citation (what things are cited at the same time), and bibliographic coupling (multiple entities citing the same thing; Boyack and Klavans, 2010). These three techniques provide options for focusing on different aspects of these networks: direct citation highlights immediate connections between information sources and use; co-citation highlights associations between similar information sources; and bibliographic coupling highlights associations among similar information consumers. Researchers have also developed a variety of techniques to enhance the nuance and insight that can be gained from citation analysis by accounting for the context surrounding citations themselves. Content-based citation analyses often include automated techniques that address and account for things such as motivations underlying citation and function a citation serves in a document (for an introduction, see Ding et al., 2014). Such techniques are valuable because citations serve at least five different types of functions for authors, and there is still debate in the field even about the extent to which citations themselves should be treated as indications of intellectual connections or social connections (Worrall and Cohn, 2023).
When used effectively, these methods can help address common limitations of citation analysis associated with a lack of nuance about why the citations appear. However, there are still several other common limitations that those involved in evaluation should consider (Worrall and Cohn, 2023). Focusing on citations narrows the scope of analysis to specific kinds of uses of information that can be captured consistently in citations and can limit the scope of assessment to only certain kinds of documents, namely those freely available online in the languages included in the analysis and indexed in a way that makes them identifiable. Inclusion of non-academic publications in citation analysis can help provide insight into non-academic impacts of information; however, including them substantially increases the time and labor necessary to perform analysis, as identification of these documents cannot currently be automated and their quality and content are less consistent than academic publications (Sibbald et al., 2015). Including these types of documents also does not fully address equity concerns about inclusion in evaluation because it leaves out audiences who do not produce documents like these or do not post them online. The study of weblinks is based on citation analysis (Thelwall, 2012), and links can be viewed as web citations akin to citations between documents (Dudek et al., 2021). However, researchers have to account for the motivations surrounding website links, as they differ from the motivations behind academic citations (Björneborn and Ingwersen, 2004).
Chapter 4 also highlights the study of social media networks, but those involved with evaluation might also benefit from additional perspectives about how this type of data can be used. Two common types of applications of social network analysis and social media are embedding learning and community learning (for a review, see Bazzaz Abkenar et al., 2021). Embedding learning models aims to determine how different nodes within social networks influence information diffusion based on a user’s characteristics and how they disseminate information. Community learning models are focused on identifying clusters of users who represent people with similar characteristics, interests, or attitudes.
Network data about event participation is also considered to be relatively easy to obtain through taking attendance or drawing it from conference call agendas or participant lists. These data can be connected with other data sources such as surveys about contacts’ close colleagues. It is still common to collect network data through questions in surveys or interviews. If the participants in the network are known in advance, it is common to develop the network map based on asking them about their relationships with one another and outcomes of interest. If the extent of the network is not known in advance, snowball sampling techniques are often used where respondents are asked to suggest subsequent respondents. For example, Cunningham et al. (2015) were able to eventually reach residents associated with the spread of climate adaptation information through a snowball sampling method that included government officials and community-based organizations. Cvitanovic et al. (2017) assessed the expanding impact of a particular knowledge broker by having them report their egocentric network every 3 months throughout the year. The broker’s perception of the strength and effectiveness of their connections was compared with that of surveyed participants in the network to assess changes in the strength of relationships over the year as well. Masuda et al. (2018) also created a research design based on known and unknown participants to assess diffusion. Working with The Nature Conservancy, they tested how sharing organizational learning workshop links with different types of people affected participation in the workshops and demonstrated that using informal contacts could enhance diffusion and changes in attitude.
This page intentionally left blank.
