APPENDIX D: WHITE PAPER ON THE INTERSECTION OF PERFORMANCE METRICS AND ATM OPERATIONS

INTRODUCTION

Performance measures are a significant factor in the planning, implementation, and continual operation of ATM programs and strategies. The use of ATM in transportation systems can often be described as a new approach that can draw scrutiny because of its relative novelty. Therefore, agencies that seek to implement ATM often have to rely on performance measures to justify the continual operation of ATM strategies or adjust various ATM strategies to meet stated program goals. Performance metrics help provide quantified evidence for how ATM strategies perform. Metrics also assist with defining the problems that ATM attempts to mitigate, establishing the overarching goals that ATM seeks to address, and assessing overall performance when starting and operating various ATM strategies.

The metrics identified within an ATM implementation program need to closely relate to the overall regional goals and objectives of the broader transportation system. Additionally, suitable performance metrics also need to rely upon the quality of available data to support the generation of those metrics. This white paper examines appropriate performance metrics for ATM operations and discusses how agencies can integrate these metrics into their transportation systems management and operations (TSMO) plans and use them to develop projects and deployments that can be evaluated to determine effectiveness.

STATE-OF-THE-PRACTICE

ATM programs typically need regular monitoring to ensure the systems and strategies meet the overall program goals. One best practice is to tie each performance metric to an overarching goal or objective to assess performance over time. The selection of performance metrics should reflect the ability of the agency to provide a narrative for how each strategy performs—especially for a nontechnical audience that may have difficulty understanding ratios, fractions, or any other complicated mathematical expressions. Specifically, suitable performance metrics should also explain how data were used to calculate and support that metric. Math should be transparent.

Important considerations are the ability to collect data, the quality of data, and the frequency of data availability. National guidelines recommend that preferable performance metrics should be SMART—specific, measurable, attainable, realistic, and time-bound. In addition, performance metrics for ATM strategies typically tend to fall into key categories based on topical goals. These topical categories usually include traffic performance, safety, public perception, sustainability, and equity.

Traffic Performance Measures

Measures for traffic performance describe the ability of a strategy or roadway to provide mobility for a transportation system. Traffic performance is a meaningful measure among the topical evaluation areas. It is measured using vehicle speed, volume, level of service, travel time savings, travel times, and travel time reliability. Vehicle speed is typically used to describe the intensity of congestion when travelers moving at slow speeds experience the direct loss of travel time savings. More holistic measures are generally captured by vehicular and person-based throughput measures that capture the extent of congestion or the proportion of a population that

experience slow speeds. Person throughput measures the number of people traveling in the corridor. Vehicular throughput has similar characteristics but only focuses on vehicles, including passenger cars, motorcycles, buses, commercial vehicles, and trucks. The level of service is a term that comes from the HCM and bases its objectives on a pre-established set of measures. For example, the HCM uses traffic density to help characterize performance on basic freeway sections.

The definition of travel time reliability focuses on the more significant effect of day-to-day variation in travel conditions not captured by the more traditional speed-based measures. Characteristics help guide the selection of travel time reliability performance measures, as described by identified goals, availability and quality of data, and geographic scale and intent of the program. Standard reliability metrics include the buffer index, planning time index, and the number of days below a pre-established threshold. The data collection method to measure speed-based performance varies by investment level. Operators typically use in-pavement loop detectors, microwave, infrared, or video sensors that can also measure other performance metrics.

Safety Measures

Many agencies currently place safety as a major priority for their transportation systems and programs, if not the main priority, particularly with the rollout and promotion of Vision Zero programs across the United States. Safety often justifies the funding and financial support of various ATM strategies for reducing conflicts, collisions, injuries, and fatalities on the roadway. Typically, agencies use historical crash data to help assess safety performance over time. One best practice is to use at least 3 years of pre-deployment data to help establish a solid statistical baseline that reduces unique seasonal variances for a proper evaluation.

Similarly, agencies should collect at least 3 years of post-deployment data to provide a fair comparison. However, gathering safety data immediately after implementing a new strategy can be problematic due to reporting delays and smaller sample sizes. Therefore, developing safety metrics may take additional time compared to the speed and congestion-based measures. Standard safety metrics for ATM strategies include the incident rate, detection response rate, clearance rate, total fatalities, serious injuries, and the number of collisions.

Public Perception Measures

Public perception performance metrics document the general awareness of various ATM strategies and the satisfaction they provide. Typical public perception performance measures include facility awareness, satisfaction, perceived value, perceived time savings, and perceived safety. Data and information for perception-based metrics usually come from surveys of facility users and the public because of the qualitative nature of this topic.

Environmental and Equity Measures

For some agencies, goals related to the environment and equity are priorities for their transportation programs, and therefore ATM strategies should reflect improvements in these areas. For an ATM strategy, specific environmental measures include metrics related to vehicular emissions (e.g., carbon monoxide, nitrogen oxide), local noise impacts, and energy and fuel usage. In addition, standard metrics that relate to equity typically capture demographic

differences within a population (e.g., income, racial, employment) as well as geographic characteristics (i.e., varied based on household and work-based locations).

CONSIDERATIONS AND IMPACTS ON ATM

A performance management system is typically just one component of an effective organizational capability for ATM operations. ATM is a relatively new concept for transportation agencies beyond the traditional approach of expanding the right-of-way and large-scale highway reconstruction projects. Because of the unique nature of many ATM concepts, agencies should assess their internal capability to deliver strategies that consider supporting technology, sound business processes, and an engaging workforce to be effective. To help evaluate organizational capacity, the FHWA developed a capability maturity model (CMM) that can assist agencies with self-assessments according to six topical categories including performance management supported by effective metrics and data. The other topical categories include: (1) business processes, (2) systems and technology, (3) culture, (4) organization and workforce, and (5) collaboration.

Within each dimension, the CMM assesses the level of capability using a numeric scale ranging from 1 to 4. Level 1 reflects agencies championing concepts and developing relationships. Level 2 reflects agencies managing staff training. Level 3 reflects agencies starting to build budgets and measuring progress. Level 4 reflects agencies with an optimized formal program.

For many agencies, the various ATM strategies operate within a TSMO program. A TSMO program includes elements of operations, planning, design, construction, maintenance, and safety, whether led by a state DOT, regional entity, congestion management agency, or some other authority. A TSMO plan helps agencies manage ATM strategies through the various project development and management processes. In addition, performance metrics can help assess agency capability and the level of success according to a TSMO plan.

Effective integration of performance metrics into a TSMO plan consists of a systematic, stepwise method that enables key stakeholders and public partners to follow along to ensure buy-in for new projects. First, the process aligns each metric to the most relevant goal (e.g., average travel speed to a congestion-related goal) to help support the rationale for pursuing the ATM strategy. Then, the integration process identifies the most appropriate data source and methodology for acquiring information and the steps used to calculate the performance measures. Performance measurement for TSMO includes defining and establishing criteria for each metric and determining how the data are received, used, and analyzed for tracking performance. Finally, the integration process describes the frequency of reporting (e.g., in real time, quarterly), the audience for the measures (e.g., policy board, public), and the continuous process and systems to support ongoing performance measurement (e.g., personnel, capital equipment, maintenance).

Formalizing that integration process into the performance management plan can help an agency align its goals with steps to achieve them. Assessing agency needs and deciding how to address those needs transparently helps streamline the process of integrating performance measurement. Additionally, data are a powerful tool that can drive discussions in many areas of an agency. Taking information previously unknown or not distributed and presenting it in a way that is easy to process or visualize can spur significant improvements and efficiencies within an agency. Acquiring the necessary data equipment and systems and configuring it optimally and efficiently is the foundation for effective performance measurement. Having robust data storage and analytic methods can improve the functionality and efficiency of a performance measurement program.

RELEVANCE TO KEY CONCEPTS

Table D-1 summarizes the relevance of performance metrics to the major topic categories of the ATM guide. This matrix helped guide the inclusion of performance metric content in the final guide.

Table D-1. Relevance of Performance Metrics to the ATM Guide Topic Categories.

OUTSTANDING ISSUES AND MAJOR GAPS

A few knowledge gaps currently exist that relate to the effective integration of performance metrics for the planning, implementation, and continuous operation of ATM. Some of these gaps stem from a lack of common understanding across various states and jurisdictions, which could improve with greater knowledge sharing and synthesis of common practices. Other gaps require a more in-depth investigation to better understand the factors most likely to contribute to success.

Developing a Priority Framework for ATM Integration Within an Agency

Agencies need additional insight into how they can prioritize ATM within their existing organization and structure. Steps to prioritize goals and performance metrics can help with the prioritization framework. Additionally, research is needed to identify effective approaches to achieving that prioritization internally—by reviewing organizational structures, business processes, and other supportive elements—prior to working with stakeholders to address the broader multimodal needs.

Identifying Best Practices for the Use of Data and Performance Metrics in ATM

Data continues to be a challenge for state agencies for a variety of reasons, including the cost of acquiring it on an ongoing basis. Best practice information is needed on cost-effective approaches for acquiring and utilizing data (in-house or from external sources), methods for integrating data into ongoing planning and operational processes, and involvement of stakeholders for ATM integration. A clear understanding of the impacts and possible limitations of data sharing agreements is also needed for effective data utilization.

Monitoring Emerging Trends and New Technologies for Use in ATM

The rapidly evolving landscape of technology and its intersection with transportation is having a disruptive impact on the use of ATM and its performance expectations from the public. Agencies need supporting research that investigates the influence of emerging technology and trends, with a focus on factors outside of the agency’s control. Such issues include but are not limited to the travel demand changes that occurred during the COVID-19 pandemic, the proliferation of electric charging stations, and the continuing development of connected vehicle technology.

Improving Organizational and Staffing Capabilities

Prior research has identified staffing shortages related to lack of technological skill as a major hurdle for the successful deployment and operation of various ATM strategies. Research is needed to identify the most effective approach for improving organizational staffing to ensure the appropriate knowledge, skills, and abilities to adequately support ATM strategies at all levels of development and implementation.

BIBLIOGRAPHY