CHAPTER 4
Implementation

Chapter 4 focuses on activities that leverage the outcomes of long-range planning. This chapter will be most useful for those looking to determine how uncertainty insights can be carried forward into action before their next plan update. Chapter sections are designed to help extend the impact of planning and maintain an agencyʼs abilities to achieve its goals. They can also support reflection, learning, and refinement for subsequent plan cycles.

4.1 Prioritization to Account for Uncertainty

One of the most concrete ways to factor uncertainty into implementation is by using information about the impacts of uncertainty to influence prioritization of funding resources for projects and programs of work. This can be achieved through multiple avenues, including:

• Analyzing projects under multiple future scenarios and elevating those that perform the best across multiple scenarios.
• Assessing differences in the effectiveness of funding or in the mix of needs across different programs of work (such as pavement, bridge, or transit) under different scenarios.
• Screening for risk or vulnerability and then prioritizing projects that respond accordingly (e.g., through adaptation, mitigation, or reduction of exposure).
• Addressing uncertainty related to funding and the environmental or planning process through “readiness”-related criteria.

The following explores each of these avenues for incorporating uncertainty into funding resource prioritization in further detail. Depending on an individual agencyʼs context, prioritization may be completed inside a long-range planning activity or as its own stand-alone process.

4.1.1 Scenarios in Project Prioritization

Quantitative prioritization processes to compare and rank projects typically have common components, including the establishment of goals, selection of evaluation criteria, selection of data and analytical approaches, weighting across metrics and objectives, analysis and ranking, and communication (Spy Pond Partners, LLC et al. 2019).

Figure 17 illustrates how an agency might go about incorporating insights from scenario planning into project prioritization.

Long Description.

The flowchart includes five main steps: Establish goals, select criteria, Data and analytical approaches; Weighting, Aggregating, and Ranking; and Communication. Each step contains specific actions:

Establish goals: Prioritize investments that address goals under multiple future conditions. Choose projects with a likelihood of success, despite uncertainty.

Select Criteria: Which metrics are sensitive to each source of uncertainty or scenario? Do any measures need to be added?

Data and Analytical Approaches: Identify models and data. Examine how they do or how they can be modified to reflect sources of uncertainty or scenarios.

Weighting, Aggregating, and Ranking: Multiple approaches, for example, Averaging scores across scenarios. Presenting scores for each scenario.

Communication: Messaging around future-proofing or being prepared. Focus on how accounting for uncertainty allows for better decisions.

4.1.2 Prioritizing Across Program Areas

Rather than ranking projects, scenarios can be used to assess differences in the effectiveness of funding or in the mix of needs across work program areas. Such an analysis would be used to inform the overall allocation of funding to programs, reflecting consideration for the uncertainty of future conditions.

4.1.3 Risk and Vulnerability

Another approach to addressing uncertainty in project prioritization is to analyze or screen for risks and then incorporate scores reflecting these risks into project scoring. Rather than focusing on the uncertainty of outcomes across different performance measures, this approach directly addresses building resilience against uncertain but potentially damaging future outcomes.

Long Description.

The bar chart illustrates Access Ohio 2045 statewide needs divided into four categories: Current Trends, Innovation, Global Markets, and Ohio Renaissance. Each category displays two bars representing ODOT and Partners. The vertical axis shows statewide needs in billions of dollars, ranging from zero to two hundred billion dollars in 50 billion dollar increments. Current Trends total 179.7 billion dollars, with ODOT at 71.7 billion dollars and Partners at 107.9 billion dollars. Innovation totals 174.4 billion dollars, with ODOT at 71.4 billion dollars and Partners at 103 billion dollars. Global Markets totals 194.1 billion dollars, with ODOT at 78.3 billion dollars and Partners at 115.9 billion dollars. Ohio Renaissance totals 191.6 billion dollars, with ODOT at 76.2 billion dollars and Partners at 115.3 billion dollars. Source: https://www.dot.state.oh.us/Documents/AO45/AO45_OhiosTransportationPlan_Final_UPDATED_110624.pdf.

4.1.4 Readiness Criteria

Since many transportation projects can take a long time to realize, project success is subject to uncertainty around future funding availability as well as the outcomes of environmental and permitting processes. To account for this, some organizations explicitly incorporate readiness criteria into project evaluation and selection. For example, in developing its TIP, the Atlanta Regional Commission collects a range of information in order to assess “project deliverability.” This includes information on the status of environmental screening and impact analysis, design, budget, and schedule (Atlanta Regional Commission 2024). Similarly, U.S. DOT evaluates discretionary funding applications for readiness (U.S. DOT n.d.).

4.2 Tracking Trends

Identifying and tracking trends upon which plan success depends can help control for uncertainties in how these trends will develop. This section describes steps for tracking trends and setting thresholds at critical points so that the agency can take action to mitigate negative effects, course correct, or realize opportunities. This section uses the term “thresholds” instead of “targets” to avoid confusion with target setting around system performance. The research team envisions tracking trends as a form of implementation that follows a long-range planning activity sought to analyze sources of uncertainty. The steps described in this section could be used in the planning process, during program and project implementation, and at regular intervals for performance reviews and updates.

This section is organized around a series of sequential steps shown in Figure 21.

4.2.1 Step 1: Define Trends

In this step, the plan team identifies the key trends upon which plan success depends and the level of uncertainty associated with each. If scenario planning, risk assessment, or similar analyses have been conducted, these may provide a starting list of notable trends and their relevance or potential effects on transportation outcomes.

Long Description.

The flowchart depicts a sequence of steps for tracking trends. It starts with Step 1: define trends, followed by Step 2: select a set of measures. The next step is Step 3: set thresholds for action, and finally, Step 4: assign responsibility and monitor performance. Each step is represented by a point on a horizontal arrow, indicating the progression from one step to the next.

Given the inherent uncertainty in forecasting, the purpose of defining trends is to understand, track, and manage changes over time. Trends of interest may affect the physical transportation system either tangibly or operationally (e.g., extreme weather events, changes in travel demand on the system as a whole or key portions of the system, or new vehicle technology and adoption), or they may affect the ability of agencies and their partners to respond effectively (e.g., priorities of elected and appointed leaders, changes in agency revenue, partnersʼ actions or inactions, or legislative changes). Tracking is a way of acknowledging that these factors can be hard to predict and may diverge from historical patterns. Moreover, if significant changes do happen, they can influence observed transportation performance and the data collected for performance management. For example, a sudden economic decline can drastically decrease ridership, which can lead to misinterpretation of performance data if not properly contextualized.

Some sources of uncertainty may be within an agencyʼs realm of influence, while others are largely beyond it. Assessing this can help determine the appropriate goals and actions related to tracking over time. In some cases, a transportation agency may seek to change the course of the trend if the agency has adequate influence (within the agencyʼs physical, financial, and legal abilities). In other cases, the goal may be to develop mitigating actions if the agency has limited influence; the goal may also be to course correct the plan or take advantage of opportunities presented by the evolution of the trend. The goal of any action should be to reinforce the agencyʼs ability to achieve its transportation goals.

To identify relevant trends to track, proceed through the following:

• Develop a list of uncertainties. These may come from existing plans and documents. They should include items directly related to the transportation system as well as the broader sociodemographic and economic context. See Section 2.2: Guided Self-Evaluation and Reflection: Sources of Uncertainty for additional support on deciding where to focus.
• Outline the degree to which agency goals and plan success may be influenced by the trend and associated uncertainty. This may be informed by qualitative or quantitative analysis and forecasting efforts conducted within a prior planning effort, as described in Section 3.3: Methods for Analyzing Uncertainty.
• Assess the level of influence that the agency has on each uncertainty, noting which mechanisms of influence are most relevant. See the “Mechanisms of Influence” sidebar for examples.

Table 30 provides a structure through which trends may be identified and provides several examples. This exercise will be most successful if it involves people with multiple areas of expertise within an agency.

Long Description.

The column headers are Trend and Associated Uncertainty, How Does Plan Success Depend on Trend, Agency’s Level of Influence, Mechanism of Influence. The data given in the table rows are as follows:

Row 1: Adoption of electric vehicles: May affect ‘gas tax’ revenue, requiring new revenue streams; Low; Support for charging infrastructure.

Row 2: Number and severity of hurricanes: May disrupt operations and require actions and spending to prepare for, manage, and repair the system after damage or disruptions; None; Indirectly, planning and investment to reduce the severity of impact.

Row 3: Average commute length; Reflects the connectivity of the transportation network and how well it aligns with community needs; Medium; Roadway network and transit service investments.

4.2.2 Step 2: Select a Set of Measures

Measures track the evolution of the trends that have been identified and can be used to evaluate the impacts of investment decisions, macro trends, and uncertainty effects after the conclusion of a given planning effort. If possible, agencies should seek to select at least one measure for each important trend. Sometimes more than one measure may be needed to describe different aspects of the trend.

The following steps support selecting measures:

• Inventory existing measures.
  • When selecting measures, it is useful to start with the measures that the agency already uses to avoid creating redundant measures and make the best use of those that are already in practice.
  • Review previous planning efforts, agency-level reporting, internal tracking and decision-making documents, and investment decisions and extract the metrics and measures used.
  • Determine if the data sources used are still available and whether any changes or additions to quality or coverage have been made.
• Assess gaps in existing measuresʼ coverage of trends.
  • Once existing measures are inventoried, assess their alignment with the identified trends and identify gaps where no existing measure corresponds with a trend.
  • Compare phenomena described by existing measures to the agencyʼs mission, goals, and objectives, as well as priority areas of uncertainty.
  • Where there are significant overlaps between measures, determine if they can be combined or made more distinct.
  • Where there are mission elements, goal areas, or objectives or priority areas of uncertainty that are not described in the existing set of measures, determine if there are new measures that could describe those elements.

• Select a set of measures.
  • Carry forward measures that continue to meet agency needs. Maintaining some level of core continuity in measures can help with longer-term monitoring and learning and support future after-action analysis.
  • Develop new measures to fill the gaps. New measures also may replace an existing measure if it is likely to be discontinued or is not adequate as a measure.
  • In selecting measures, consider the factors of successful tracking measures shown in Figure 22.

Long Description.

The infographic presents six characteristics of successful tracking measures. Each characteristic is represented by an icon and a brief description. ‘Implementable’ is shown with a gear and pencil icon, and says implementable within agency financial, technical, and staff resources. ‘Closely related’ features a network icon, and says closely related conceptually to the corresponding trend. ‘Meaningful’ is depicted with a head and lightbulb, and says meaningful to decision makers. ‘Easy to communicate’ uses a speech bubble, and says easy to communicate and to understand. ‘Sensitive’ is represented by a shield with a check mark, and says sensitive to policies or other factors that might affect the trend. ‘Updateable’ is shown with a circular arrow, and says updateable with future data available on a time cycle that supports decision making.

4.2.3 Step 3: Set Thresholds for Action

This section discusses how to set thresholds that cause an agency to take action. These thresholds often represent an amount of change in the trend that could threaten the agencyʼs ability to fulfill its mission or achieve the future envisioned by the plan. Conversely, they could represent new opportunities becoming available. These thresholds may align with targets that agencies set under state or federal regulations, but they will often be different since the thresholds do not necessarily represent a desired outcome.

The following steps support setting thresholds:

• Conduct analysis that provides decision-makers with the context for the possible futures the agency may face. Methods may include the following (see Section 3.3: Methods for Analyzing Uncertainty for more detail):
  • Scenario planning. Conduct analysis on a set of discrete alternative future scenarios representing different configurations of the environment within which the agency will be working (e.g., demographic or economic changes, new technology adoption, federal policy or funding changes, or environmental changes).
  • Exploratory modeling and analysis. Use a systematic approach to sensitivity analysis on broad ranges for each input. Identify patterns across many potential futures to guide decision-making toward choices that will perform the best in the greatest number of possible future circumstances.
  • Stakeholder or expert feedback. When quantitative data is not available, or its frequency or detail is low, qualitative information can be gathered from stakeholders, experts, or even groups made up of the general public. Their assessment of conditions and their progression over time can help fill in gaps in more quantitative datasets.
• Assign thresholds for each measure. These thresholds delineate when data measurements would catalyze agency action and what that action may include. For example, when fatalities rise by a certain percentage year over year, the agency may reevaluate its safety investment strategies. The threshold should include:
  • A quantitatively determined value (could be a number, percent, or rate of change) and whether it is an upper or lower bound.
  • A description of how the value was chosen and what it is intended to signify.
  • Notes on data sources and review frequency.
• Decide what actions to take if the threshold is crossed.
  • The actions should allow the agency to respond to or mitigate a situation to keep the plan on track or to take advantage of new opportunities.
  • Appropriate actions might include changing levels of funding targeted at a particular issue, fast-tracking a specific analysis or strategy, reprioritizing a list of projects, updating relevant agency policies, or convening a “response team” of experts or partners.

4.2.4 Step 4: Assign Responsibility and Monitor Performance

Now that thresholds have been set, the agency should follow through with tracking the trends to make sure that the associated measures do not cross the thresholds or, if they do, initiate the previously decided changes or actions.

The following steps support the assignment of responsibilities and monitoring of performance:

• Assign responsibilities to monitor measures, ensuring that relevant management is aware of the distribution of those responsibilities. The office or offices within the agency that are assigned responsibility should be able to:
  • Collect data from many parts of the agency and from outside sources.
  • Have the technical ability to use the data to calculate measures and track their changes over time.
  • Communicate with agency leaders and decision-makers so that they are notified if thresholds are crossed.
  • Share the actions that the plan has specified should thresholds be crossed.
• Begin tracking metrics for the new performance measures set.
  • When responsibility is assigned, determine a frequency for updating the measures and reporting results and identify a venue in which the information will be shared.
  • Trends that change more slowly can generally be reported less frequently than trends that change more quickly.
• If the trend moves beyond the threshold, it should initiate the actions previously selected by the agency.

Table 31 shows a table filled in with example measures that can guide an organization through gathering the information needed to designate these trends and thresholds.

Table 32 provides examples of other agenciesʼ tracking metrics, including a brief description, the measure being tracked, what action(s) the agency associated with trends in that measure, and links to source material for further reading. Note that this research did not identify any agencies that had set specific thresholds for transportation measures that would trigger specific actions by the agency or its partners, but that setting thresholds for action would be an extension of current approaches to measures tracking.

4.3 After-Action Analysis

After-action analysis can help an agency better manage uncertainty over time by building in mechanisms of feedback and learning into agency planning. After-action analysis can reveal opportunities to enhance the effectiveness of agency planning by examining follow-through and providing insight into whether strategies and actions had their intended effects.

This section addresses two main components of after-action analysis: (1) plan implementation, in which actions are taken and then monitored based on existing strategies or policies and (2) ex-post analysis, in which the planʼs effects and effectiveness are analyzed.

4.3.1 Plan Implementation

Implementation plans provide a mechanism for coordinating steps that should take place following plan completion. By providing additional specificity, accountability, and a plan for tracking, implementation plans can reduce the gap between “what we said we would do” and “what actually gets done.” This is important for managing uncertainty both because implementation is key to any strategyʼs success, including those related to addressing uncertainty, and because agency capacity and follow-through over time is itself a source of uncertainty around the effectiveness of plans.

Implementation plans flow out of the rest of the planning process and are informed by a planʼs vision, goals, and objectives; strategies produced through plan analysis; other existing plans; and public and stakeholder involvement. These efforts produce a series of proposed actions, policies, and strategies.

The following steps support plan implementation:

1. Inventory the proposed actions, strategies, and policies that the plan has developed. Sometimes it is useful to group actions by the agency policy or strategy that they support.
2. Assign responsibility for each action to a person, team, or office within the agency.
3. Describe other key characteristics to help guide implementation, such as a priority level, key implementation partners, a rough implementation timeline (e.g., 1–4 months), and approximate cost (either monetized or qualitatively assessed as “high,” “medium,” or “low”).
4. Regularly track and enforce implementation, ideally at a high level within the agency and using the timeline prescribed in the implementation plan.

Table 33 shows a structure that can serve for organizing implementation actions.

Engaging with uncertainty may also influence the structure within which actions are developed. An example of this is outlined in the text box that follows.

Long Description.

The column headers are: Action ID, Description of Action, Responsible Party, Key Partners, Timeline, Priority. The data given in the table-wise rows are as follows:

Row 1: T-1: Develop procedures for collecting and processing data on trends to be reported; Planning Division; Strategic Initiatives Team, IT Division; Within 4 months of plan completion; Medium.

4.3.2 Ex-Post Analysis

Ex-post analysis is a quantitative analysis of desired outcomes, looking backward in time to identify the impacts or portions of impacts that resulted from an agencyʼs plans, projects, or actions. Agencies collect data about outcomes of multiple intentionally selected plans, projects, or actions to determine whether they produced the desired benefits or had unintended effects. Examining this set of plans, projects, and actions can reveal areas of uncertainty that caused outcomes to deviate from what was expected, allowing the agency to determine how to better account for these uncertainties in the future.

The following steps support ex-post analysis:

1. Define the objective of performing ex-post analysis. Determine if there is a specific aspect of planning that the agency is seeking to improve, such as forecasting accuracy or project selection.
2. Select projects or policies to evaluate. Select a sample of past plans, projects, or agency actions that will be evaluated within the ex-post analysis. These should be related to the objective from Step 1 and ideally cover a range of geographies, project sizes, and types.
3. Assess data maturity levels. Ensure that the data about these projects, plans, or actions is complete, available, and detailed enough to perform the analysis. This includes inventorying the data that is available. The types of data shown in Figure 23 may be needed often. If the data are not adequate for ex-post analysis, establish processes to collect better data in the future.
4. Perform ex-post analysis. Use the gathered data to conduct a thorough retrospective analysis of each project or policy. Evaluate the accuracy of initial forecasts and assumptions, assess the factors that contributed to project success or failure, and identify any sources of uncertainty that were not adequately accounted for during the planning phase. Changes in performance should be related to actions taken because of a plan or project rather than general variations caused by external factors. Thus, it is important to recognize and account for external factors.

   Statistical analysis may help to evaluate the change in performance and outcomes and identify causative and correlative links. However, oftentimes an impact cannot be isolated to a

Long Description.

The chart outlines four categories of information essential for ex-post evaluation: work completed, anticipated benefits, project context, and measuring outcomes. ‘Work Completed’ asks: Are there adequate scopes to categorize projects? Is there good historically archived information about what was actually done? ‘Anticipated Benefits’ asks: What were the expected outcomes? Are there plans or analyses of anticipated benefits? ‘Project Context’ asks: What external factors could have affected outcomes? Are there data on these so that the effect of the action can be isolated? ‘Measuring Outcomes’ includes subjective assessment, cross-sectional data (one-time observations) comparing treated to untreated locations, and longitudinal data showing before and after changes.

3. single cause. Therefore, effects should be evaluated at the network or community scale rather than the facility or individual scale.

   If the analysis is occurring at the programmatic level, this is an opportunity to examine relationships between funding allocation, project performance, and outcomes. These analyses evaluate questions such as whether funding was distributed in accordance with agency goals or if budget constraints or cost overruns affected outcomes. Analysis at the program level may also compare the impacts of modal splits on community outcomes (e.g., did a roadway or transit project have a greater positive influence on quality of life?) or the degree to which investment aligned with other entitiesʼ resource allocations (e.g., were the DOT and MPO able to take advantage of investment collocation temporally or spatially?).

4. Identify lessons learned. Identify the trends and patterns in plans, projects, or actions realizing their expected benefit while others did not. What could have been done differently to better predict the actual effects of the plan, project, or action? This process will often require expertise and input from different divisions or individuals within an agency.
5. Make suggestions for accounting for uncertainty in the future. What changes could the agency make to better account for uncertainty? For example, are there changes in modeling methodologies, decision-making processes, forecasting, project prioritization, or implementation strategies?