Developing a Strategy to Evaluate the National Climate Assessment (2024)

Chapter: Appendix C: Crosswalk Between Statement of Task and Overarching Evaluation Questions

Visit NAP.edu/10766 to get more information about this book, to buy it in print, or to download it as a free PDF.
Previous Chapter: Appendix B: NCANet
Page 109 Cite Bookmark
Suggested Citation: "Appendix C: Crosswalk Between Statement of Task and Overarching Evaluation Questions." National Academies of Sciences, Engineering, and Medicine. 2024. Developing a Strategy to Evaluate the National Climate Assessment. Washington, DC: The National Academies Press. doi: 10.17226/27923.
Save
Cancel

Appendix C

Crosswalk Between Statement of Task and Overarching Evaluation Questions

Table C-1 below provides a map connecting the themes and questions provided in the Statement of Task to the preliminary overarching evaluation questions presented in Chapter 3. As can be seen in the table, all of the questions that appear in the Statement of Task have one or more corresponding overarching evaluation questions or subquestions. It is also important to note that while there is very close alignment between the Statement of Task questions and the preliminary overarching evaluation questions, there are some differences in framing. This reflects the committee’s decision to align the evaluation questions with the preliminary logic model. Throughout the evaluation design process carried out by USGCRP and their consultant, the committee anticipates that these questions will continue to evolve as a more refined logic model is developed that reflects an updated understanding of the program and the priorities of the key evaluation users. There are several overarching evaluation questions or subquestions presented in this report that do not correspond to the questions in the Statement of Task. Specifically, the committee added question 5 (“How do the contextual factors described in the logic model influence how audiences feel, what they gain (e.g., cognition), what they do (e.g., behaviors), and how do they mediate the use of the NCA?”) and subquestion 5a (“How did the characteristics of the audience—including those who have been historically disenfranchised—affect the extent to which their informational and decision-making needs were met?”). This decision reflects the emphasis in the Statement of Task that the committee consider diversity, equity, inclusion, and justice (DEIJ) principles.

In addition, as described in Chapter 4, the committee finds that exploring the network of networks approach is critical to understanding the mechanisms through which the NCA and other USGCRP products are used. As such, the committee introduced evaluation question 6 (“How does the network of networks factor into use?”) This will illuminate the Statement of Task themes with regards to usefulness, decision making, and awareness/engagement. However, because the committee’s framing is specific to the network of networks concept, those questions are not presented in the table.

Page 110 Cite Bookmark
Suggested Citation: "Appendix C: Crosswalk Between Statement of Task and Overarching Evaluation Questions." National Academies of Sciences, Engineering, and Medicine. 2024. Developing a Strategy to Evaluate the National Climate Assessment. Washington, DC: The National Academies Press. doi: 10.17226/27923.
Save
Cancel

TABLE C-1 Crosswalk Between Themes and Questions from the Committee’s Statement of Task and Preliminary Overarching Evaluation Questions Presented in Chapter 3

Statement of Task Themes Statement of Task Questions Preliminary Overarching Evaluation Questions or Subquestions
Regarding usefulness How and to what extent have stakeholders found NCA materials to be useful? What specifically has been useful?
  • How, and to what extent, did NCA products address information needs among priority audiences (i.e., what did they gain cognitively in terms of knowledge, skills, attitudes, capacities, etc.)? (Q2)
  • Which products or parts of products contributed to meeting information needs? (Q2b)
  • How, and to what extent, did NCA products address decision needs among priority audiences (i.e., what did they do as a result of using the products)? (Q3)
  • Which products or parts of products contributed to meeting decision needs? (Q3b)
  • How did the characteristics of the audience—including those who have been historically disenfranchised—affect the extent to which their informational and decision-making needs were met? (Q5a)
Does the selection of topics, regions, sectors, and level of detail (e.g., time frame, spatial resolution, subsector concerns) in the NCA (and USGCRP products it references as well as related USGCRP products) adequately address the stakeholders’ needs?
  • How does the selection of topics, regions, sectors, level of detail, ways of expressing uncertainties, and other aspects of what information is included and the way it is organized and presented contribute to audiences’ perceptions of how the products meet their needs? (Q4b)
What decision or informational needs were well-addressed by the NCA? What decisions can stakeholders make given the level of information provided?
  • What informational and other cognitive needs were well addressed by the NCA? (Q2a)
  • What decision needs were well addressed by the NCA? What decisions can audiences make given the level of information provided? (Q3a)
Regarding decision-making, future needs, and missing information, details, and/or tools What future needs are anticipated? What additional types of decisions (if any) do stakeholders anticipate they would revisit given different topics and/or levels of detail? What information would be required to meet needs that USGCRP is not meeting already?
  • Given the evolving nature of the field, what are future needs and how could USGCRP help meet them? (Q5c)
What decision or information needs did the stakeholders expect would be met by the NCA but were not?
  • What wasn’t gained that could or should have been? What are the gaps in what the NCA provided and what else did potential audiences use to cover them (including sources based on USGCRP products)? Are these gaps that audiences expect the NCA to meet? (Q2c)
  • What actions weren’t taken because audiences did not have their decision needs adequately met? What are the gaps in what the NCA provided and what else did potential audiences use to cover the gaps or for other purposes (including sources based on USGCRP products)? Are these gaps that audiences expect the NCA to meet? (Q3c)
  • Given the abundance of other climate change resources, what is the unique value of the NCA and related products? (Q4c)
  • How did the context of multiple information sources affect the impact of NCA? Did they expand its impact? Did they replace it? (Q5b)
Page 111 Cite Bookmark
Suggested Citation: "Appendix C: Crosswalk Between Statement of Task and Overarching Evaluation Questions." National Academies of Sciences, Engineering, and Medicine. 2024. Developing a Strategy to Evaluate the National Climate Assessment. Washington, DC: The National Academies Press. doi: 10.17226/27923.
Save
Cancel
Statement of Task Themes Statement of Task Questions Preliminary Overarching Evaluation Questions or Subquestions
For stakeholders whose decision or informational needs were not met by the NCA and selected USGCRP products, what is the reason? What other products/materials, including other USGCRP and non-USGCRP products, did they use, if any?
  • What wasn’t gained that could or should have been? What are the gaps in what the NCA provided and what else did potential audiences use to cover the gaps or for other purposes (including sources based on USGCRP products)? Are these gaps that audiences expect the NCA to fill? (Q2c)
  • What actions weren’t taken because audiences didn’t have the info that was needed? What are the gaps in what the NCA provided and what else did potential audiences use to cover the gaps or for other purposes (including sources based on USGCRP products)? Are these gaps that audiences expect the NCA to meet? (Q3c)
Stakeholder awareness and engagement How aware or involved were different stakeholder groups in the NCA development process and how did this influence their use of the report? For stakeholders who were not previously aware of the NCA development process, how did they become aware of the NCA?
  • To what extent are priority audiences aware of NCA products and what are the most effective ways to increase awareness? Did involvement in the development process contribute to general awareness and use of the report? (Q1)
  • For audiences who were not previously aware of the NCA, how did they become aware of the NCA? What factors, including the NCA development process, are related to awareness of the NCA? (Q1a)
  • How, and to what extent, did users share information from NCA products with others? What could be done to make it easier for users to push information out to others? (Q1b)
  • To what extent are audiences using other resources that are based on USGCRP products? (Q1c)
  • What groups have been involved in the NCA development process and to what extent were they involved? How effectively were historically marginalized and underrepresented stakeholders included? (Q1d)
  • How did engagement in the process affect use of the NCA? How did the engagement of historically marginalized communities and underrepresented audiences affect use of the NCA? (Q4a)
How effectively does the NCA development process engage historically marginalized communities and underrepresented stakeholders?
  • How did engagement in the process affect use of the NCA? How did the engagement of historically marginalized communities and underrepresented audiences affect use of the NCA? (Q4a)
  • How did the characteristics of the audience—including those who have been historically disenfranchised—affect the extent to which their informational and decision-making needs were met? (Q5a)
How understandable and navigable is the NCA, including the report documents and findings, and underlying supporting data? Is the NCA information presented in a format that informs decision-making?
  • How did the attributes of the products and process contribute to how users feel, what they gain (e.g., cognition), and what they do (e.g., behaviors)? What about the products and process could be changed to make them more effective? (Q4)
  • How does the selection of topics, regions, sectors, level of detail, ways of expressing uncertainties, and other aspects of what information is included and the way it is organized and presented contribute to audiences’ perceptions of how the products meet their needs? (Q4b)
Page 112 Cite Bookmark
Suggested Citation: "Appendix C: Crosswalk Between Statement of Task and Overarching Evaluation Questions." National Academies of Sciences, Engineering, and Medicine. 2024. Developing a Strategy to Evaluate the National Climate Assessment. Washington, DC: The National Academies Press. doi: 10.17226/27923.
Save
Cancel
Statement of Task Themes Statement of Task Questions Preliminary Overarching Evaluation Questions or Subquestions
Items added based on the logic model
  • What organizations or types of organizations are the most powerful in spreading USGCRP products to other audiences (i.e., most influential nodes in the network)? What are the approaches that best support the sharing of information through the network of networks? (Q6a)
  • How does USGCRP instruct or amplify the work that is being done by agencies and other partner organizations? How does the information generated by the NCA help facilitate their work? (Q6b)
  • Which part of this effort directly involves USGCRP vs what is appropriate for the extended network? (Q6c)
  • How did NCA affect other information sources regarding climate change? In other words, do those information sources make use of it? (Q6d)
  • Has NCA affected the degree to which researchers and decision-makers collaborate? How does NCA affect the identification of research needs or development of additional research? (Q6e)
  • Who is left out in the network of networks approach? (Q6f)
  • How do frontline federal workers in these agencies feel, what do they gain, what do they do as a result of the NCA? (Q6g)
Page 109 Cite Bookmark
Suggested Citation: "Appendix C: Crosswalk Between Statement of Task and Overarching Evaluation Questions." National Academies of Sciences, Engineering, and Medicine. 2024. Developing a Strategy to Evaluate the National Climate Assessment. Washington, DC: The National Academies Press. doi: 10.17226/27923.
Save
Cancel
Page 109
Page 110 Cite Bookmark
Suggested Citation: "Appendix C: Crosswalk Between Statement of Task and Overarching Evaluation Questions." National Academies of Sciences, Engineering, and Medicine. 2024. Developing a Strategy to Evaluate the National Climate Assessment. Washington, DC: The National Academies Press. doi: 10.17226/27923.
Save
Cancel
Page 110
Page 111 Cite Bookmark
Suggested Citation: "Appendix C: Crosswalk Between Statement of Task and Overarching Evaluation Questions." National Academies of Sciences, Engineering, and Medicine. 2024. Developing a Strategy to Evaluate the National Climate Assessment. Washington, DC: The National Academies Press. doi: 10.17226/27923.
Save
Cancel
Page 111
Page 112 Cite Bookmark
Suggested Citation: "Appendix C: Crosswalk Between Statement of Task and Overarching Evaluation Questions." National Academies of Sciences, Engineering, and Medicine. 2024. Developing a Strategy to Evaluate the National Climate Assessment. Washington, DC: The National Academies Press. doi: 10.17226/27923.
Save
Cancel
Page 112
Next Chapter: Appendix D: Network Analysis: Additional Measures and Strategies
Subscribe to Email from the National Academies
Keep up with all of the activities, publications, and events by subscribing to free updates by email.