Appendix E
Testing Peer Exchange Summary

Introduction

This appendix summarizes the key points from the NCHRP 08-154 Peer Exchange held on September 30, 2024. The peer exchange was the last element of Task 6, which aimed to test and support refinement of a guide designed to help state DOTs and MPOs integrate uncertainty into long-range transportation planning.

Task 6 proceeded in three steps: 6A) Development of a testing plan, 6B) Agency testing and validation, including workshops and 6C) a web-based peer exchange to share lessons learned from test sites, along with participation and interaction from a wider audience.

The main goals of the peer exchange were to:

• Invite sharing by each of the test agencies on their experience with guide components, including lessons learned and identification of future opportunities
• Facilitate knowledge sharing among agencies on their approaches to incorporating uncertainty into long-range planning
• Gather feedback to strengthen the guide, especially through practical examples
• Gather input on the guide’s applicability in diverse agency settings
• Identify interest or need for further research or implementation efforts beyond the project

Following this peer exchange, the research team is now pivoting to focus on guide refinements, including:

• Response to previously received Panel comments
• Development of summary and introductory material to the guide to help readers understand the goals and benefits of addressing uncertainty and how to navigate to the guide content that is most relevant to a particular use.
• Adjustments to individual guide components based on applications in the agency testing (for example: adjusting templates, descriptions, or terminology)
• Incorporation of illustrative examples from testing into the guide

The peer exchange included participation by staff from each test agency, the project panel and program officer, and the research team.

Key Insights/Themes

Even when uncertainty has been a topic of discussion and analysis at an agency, it is still a daunting task to effectively and rigorously integrate into planning practice. The test agencies found the framework presented by the guide helpful and provided feedback to increase its utility for practitioners. The following themes emerged from agency presentations and discussion:

• Uncertainty planning involves a complex web of relationships, and addressing that web requires extensive internal coordination.
• Communicating about uncertainty will require thoughtful effort to enable better decisions rather than muddying the water.
• Agencies will need more data and expertise from a wide range of sources to understand which factors they can control, and which are at the mercy of broader trends.
• Smaller agencies will likely need data, expertise, and framework support to be able to integrate uncertainty into their current practices.

While this project’s guide does begin to address these challenges, it must also make a concrete and specific case to agencies about why they should engage with uncertainty, how it will help in what they are already doing, what they should be prepared for, and how doing so will positively impact the communities they serve.

This conversation with practitioners also highlights the need for more extensive implementation efforts to facilitate best practices in integrating uncertainty planning into complex systems. Not only was there significant discussion about how to communicate these ideas to decision makers and the public, but participants also asked questions of their peers about internal communication, data planning, and forecasting of external trends that they have otherwise not engaged with. Some agencies are ready and interested in working on this topic, but many struggle to focus in on useful and targeted activities given limited resources (whether that be staff, data, or time) to do so. Several attendees noted that there have been several attempts made over the years at better understanding uncertainty’s impact, only for them to fall by the wayside after a specific planning effort is completed. Therefore, further work is needed to connect exploration or analysis of uncertainty to how these activities can benefit agencies and support ongoing decision-making and agency management. That is, uncertainty cannot be considered as a “one-and-done” topic for a stand-alone plan, but rather should be considered within an ongoing iterative cycle of planning, learning, implementation, and feedback.

Agency Presentations

Five agencies - Atlanta Regional Commission (ARC), Iowa DOT, MassDOT, VTrans (Vermont), and Kentucky Transportation Cabinet (KYTC) - shared their experiences and insights from testing the

NCHRP Research Report 1168 guide. The slides presented by each agency and used by the research team in facilitation are included as an appendix to this memo.

1.1. Atlanta Regional Commission

1.1.1. Testing Focus

ARC concentrated on the “Family of Plans” and “Communications Playbook” components of the NCHRP Research Report 1168 guide. The agency aimed to find ways to better coordinate and connect the diverse array of plans they manage and develop flexible outreach strategies related to uncertainty in transportation planning.

1.1.2. Key Challenges

ARC shared several challenges they have faced in prior engagement with uncertainty and in the testing of the guide:

• Uncertainty planning was often perceived as a novel and abstract concept within the organization. This has sometimes prevented efforts from even starting or from getting off the drawing board.
• The workflow for addressing uncertainty in past plans and modeling efforts was unwieldy, involving a lot of trial and error.
• There is insufficient policy direction from ARC’s policy board regarding uncertainty planning, and recent direction from organizational leadership has indicated that the boards should be leading policy.
• The agency recognizes a need to engage the general public effectively on uncertainty-related issues but have yet to find a suitable platform for doing so.

1.1.3. Takeaways

Through their engagement with the guide, ARC gained the following insights:

• Managing numerous “individual plans” as part of a “Family of Plans” concept would be a significant challenge but doing so would also present benefits beyond uncertainty planning in the form of better coordination between groups and a reduction of duplicated work and analyses.
• The sheer number of plans might prove too difficult to manage effectively for incorporating decision-making under deep uncertainty (DMDU). Uncertainty may need to be addressed starting with specific individual efforts.
• Future individual plans should be more deliberately interconnected to enhance coordination and effectiveness.
• Visualizing plan cycles and schedules in a centralized location provided valuable clarity.
• Addressing uncertainty is crucial for resource allocation, but also a daunting task.

• Elected officials and policymakers tend to be more receptive to information presented with certainty, posing a communication challenge.

1.1.4. Future Plans

ARC outlined the following steps for integrating uncertainty into their long-range plans:

• Design a process and work program to incorporate DMDU into the next Metropolitan Transportation Plan (MTP) update due in 2028. This process will involve:
  • Determining all potential “points of entry” for incorporating DMDU and selecting the most suitable ones.
  • Utilizing additional activities and resources provided in NCHRP Research Report 1168, including the Self-Evaluation tool.
  • Expanding the scope to include plans typically leveraged during the MTP update process.
  • Leveraging the updated Community Engagement Plan to actively include the general public in planning for uncertainty.

The guide provided ARC with a framework to design a more effective scenario planning and uncertainty planning work program, which they plan to use in the development of their next MTP update. ARC also plans to leverage this process as a mechanism to incorporate their emerging climate work with their traditional transportation planning process.

1.2. Iowa DOT

1.2.5. Testing Focus

With their next long-range planning cycle approaching, Iowa DOT concentrated on the “Methods for Analyzing Uncertainty” and “Tracking Trends” guidance offered in NCHRP Research Report 1168. The agency’s goal was to explore how to integrate uncertainty considerations into the development of their next State Long-Range Transportation Plan (SLRTP) and Transportation Asset Management Plan (TAMP), both due for updates in 2027.

1.2.6. Key Challenges

Iowa DOT highlighted these key challenges:

• Finding Value in Scenario Planning: The agency grappled with the challenge of translating scenario planning exercises into concrete, actionable strategies and sought other methods to engage with uncertainty.
• Adapting Uncertainty Planning to a Rural Context: Applying uncertainty planning concepts to their largely rural, agricultural state presented unique considerations that are not as well explored as conditions in more urbanized regions.

• Distinguishing Uncertainty from Risk Management: Understanding the relationship between uncertainty planning and traditional risk management, and how these approaches complement each other, was also a point of discussion.

1.2.7. Takeaways

Iowa DOT found the following takeaways from guide testing particularly insightful:

• Uncertainty planning does not have to follow a rigid, one-size-fits-all approach. There are multiple ways to incorporate uncertainty considerations into their existing planning processes and documents.
• It’s important to strike a balance between considering external factors (which they cannot control) and internal factors (which they can influence).

1.2.8. Future Plans

Building upon their workshop experiences, Iowa DOT plans to:

• Build upon the activities from the workshop in the development of their next SLRTP and TAMP.
• Expand their uncertainty mapping exercises beyond pavement and bridge risks to encompass other topics, such as capacity planning.
• Explore the integration of more leading indicators into their planning and performance/trend tracking, supplementing the lagging indicators they have traditionally relied upon. This shift reflects a more proactive and forward-looking approach to planning.

1.3. MassDOT

1.3.9. Testing Focus

MassDOT, building on prior work in integrating uncertainty into transportation planning, chose to focus on specific action items related to resiliency outlined in their “Beyond Mobility” long-range plan. In their testing workshop, they specifically homed in on:

• Building the Future Workforce: Addressing the need to develop a skilled workforce capable of tackling the challenges of uncertainty.
• Training for Uncertainty: Equipping their staff with the knowledge and skills necessary to navigate uncertain futures.
• Managing and Sharing Data: Emphasizing the importance of robust data management and sharing practices in the context of uncertainty planning.

1.3.10.Key Challenges

MassDOT identified several key challenges in understanding and managing uncertainty:

• Workforce Capacity: Ensuring their workforce has the capacity to effectively implement the ambitious actions detailed in their “Beyond Mobility” plan, such as fare-free transit, EV equipment manufacturing, and enhanced safety measures for the MBTA.
• Federal Funding: The unpredictable nature of future federal funding levels for transportation projects.
• Interjurisdictional Cooperation: The need for seamless cooperation across different jurisdictions, particularly between MassDOT and municipalities, to realize the vision outlined in “Beyond Mobility.”

1.3.11.Future Plans

Implementing “Beyond Mobility” Actions: MassDOT is likely to prioritize the implementation of action items related to resiliency outlined in their “Beyond Mobility” plan including building a future-ready workforce, providing training to enhance their capacity for dealing with uncertainty, and improving data management and sharing practices.

1.4. VTrans (Vermont)

1.4.12.Testing Focus

VTrans chose to test the “Self-Evaluation Score Sheet” and “Family of Plans” sections of the guide. These sections aligned with their goals to:

• Facilitate internal conversations about sources of uncertainty and their potential impact.
• Identify opportunities to better leverage and coordinate their existing planning efforts within a “family of plans” framework.

1.4.13.Key Challenges

As a smaller agency, VTrans has limited staff resources, making it difficult to dedicate sufficient time and personnel to undertake comprehensive uncertainty planning.

1.4.14.Takeaways

VTrans had several key takeaways from their experience:

• Value of External Facilitation: They found it beneficial to have neutral, respected outsiders facilitate the workshop and help tailor the guide’s content to their specific needs.
• Benefits of Cross-Divisional Engagement: The workshop fostered valuable cross-divisional awareness and understanding of uncertainty-related work that had previously been siloed within different departments.
• Need for a More Inclusive “Family of Plans” Approach: VTrans provided constructive feedback on the “Family of Plans” worksheet, suggesting it should allow for more flexibility, encompass a wider range of plans, and be redesigned for enhanced usability.

1.4.15.Future Plans

VTrans plans to:

• Hold annual uncertainty/risk workshops: They will build on the momentum from the NCHRP workshop by holding annual internal workshops to further integrate uncertainty and risk considerations into their planning.
• Use TAMP and LRTP Updates as Opportunities: They aim to use future Transportation Asset Management Plan (TAMP) and Long-Range Transportation Plan (LRTP) updates as opportunities to delve deeper into enterprise risk management.
• Expand Risk Assessment Efforts: They are expanding their use of the Highway Transportation Resilience Planning Tool (TRPT) to include railroad corridors, with the goal of developing a new risk chapter for their Rail Plan.
• Consider a Public Opinion Survey: They are considering conducting a public opinion survey as part of their LRTP update process to gain a better understanding of public sentiment, particularly regarding strategic disinvestment.
• Address Challenges of Follow-Through: VTrans acknowledges the difficulty of maintaining momentum for change and ensuring that the insights gained from workshops like this translate into sustained action, particularly given their limited resources.

1.5. Kentucky Transportation Cabinet

1.5.16.Testing Focus

Relatively new to incorporating uncertainty into their planning, KYTC focused on their existing Long-Range Statewide Transportation Plan (LRSTP), which was their first foray into scenario planning. They used the workshop as an opportunity to examine this plan through the lens of four key uncertainties:

• Funding: Addressing the uncertainties related to securing and allocating funding for transportation projects.
• Resiliency: Preparing for and mitigating the impacts of disruptions, whether from natural disasters, or other unexpected events.
• Technology: Considering the potential influence of rapidly evolving technologies on transportation systems and planning.
• Freight: Addressing uncertainties related to freight movement and logistics.

KYTC engaged with the “tracking trends” and “after action analysis” components of the guide.

1.5.17.Key Challenges

KYTC’s testing and analysis brought to light several key challenges:

• Developing an Adaptive Transportation System: Creating a transportation system that is resilient and can adapt to potentially disruptive forces, such as rapid advancements in technology, funding challenges and constraints, human-made and natural disasters.
• Navigating Legislative Constraints: The Kentucky Legislature exerts significant control over the state’s highway plan, often focusing on a phase-by-phase approach. This can make it challenging for KYTC to pursue project-based resiliency efforts that require a more long-term, consistent approach.

1.5.18.Takeaways

KYTC highlighted the following takeaways from their experience:

• Importance of a Structured Approach to Uncertainty: The guide’s emphasis on a clear, structured approach to uncertainty planning resonated with KYTC. This approach involves:
  • Defining the Uncertainty: Clearly articulating the specific uncertainties they are seeking to address.
  • Selecting a Measure: Identifying appropriate metrics to track and monitor the uncertainty.
  • Projecting or Monitoring: Developing projections or implementing monitoring mechanisms to track how the uncertainty evolves.
  • Creating an After-Action Plan: Establishing a plan for responding to different outcomes or changes in the uncertainty.
• Value of Leadership Engagement: KYTC found it beneficial to involve key leaders from various divisions within their organization. Engaging leadership helps ensure diverse perspectives are considered and facilitates a more comprehensive approach to uncertainty planning.

1.5.19.Future Plans

KYTC plans to:

• Incorporate Uncertainty into Future Plans: They intend to integrate uncertainty considerations into the development of their future LRSTPs.
• Pursue Actionable Steps Based on Funding: Implement specific, actionable steps to address uncertainty based on funding availability.
• Use Risk-Based Cost Estimates: They are exploring the use of risk-based cost estimates earlier in the project development process to mitigate potential cost overruns stemming from known uncertainties. This includes considering ranges of cost estimates that can be refined as projects progress.

• Develop a Family of Plans: They plan to leverage the guide to develop a more cohesive “family of plans” to guide decision-making. This will help ensure greater alignment between their various plans and create a more adaptable and resilient transportation system.

Group Discussion

The concluding discussion session provided a platform for open dialogue and reflection. Participants shared feedback on the guide and emphasized the need for clearer policy direction, suggesting strategies for effective public engagement, and highlighting the importance of addressing the financial and logistical constraints faced by smaller agencies.

The “Family of Plans” concept, while well-received, prompted suggestions for improvement. Agencies recommended a more inclusive approach, incorporating a broader range of plans within the worksheet, and a structural overhaul to enhance its user-friendliness. The discussion also highlighted a preference for a concrete-to-abstract learning progression, advocating for the introduction of uncertainty concepts through relatable examples, such as a washed-out road, before venturing into more abstract topics like funding models or emerging technologies.

Participants suggested incorporating more storytelling and practical examples to effectively communicate the importance and relevance of uncertainty planning to a wider audience. The exchange also underscored the importance of sustainable engagement beyond the workshop and project, with suggestions for courses, follow-up initiatives, and even the development of an online platform to showcase “Family of Plans” examples as valuable resources.

Participants emphasized the importance of continued learning in understanding and managing uncertainty. Given the complexity of the topic and its relevance across many areas of planning and implementation, it is important to set expectations on gradually building capacity over time. As agencies work to develop and implement processes related to uncertainty, they would benefit from ongoing mentorship or more opportunities to talk to researchers and peers.