CHAPTER 3

Results of Surveys with State Departments of Transportation

Introduction

The research team developed a survey instrument to gather input from state agencies on their use of Machine Learning (ML) methods and applications. The survey questions were created to capture as much information as possible from each transportation agency not only on their current use of ML but also on their plans for increasing the adoption of ML in their applications. The questions also attempted to capture the challenges, obstacles, or limitations some agencies may have experienced in the development of current applications.

A web-based survey was launched by Old Dominion University (ODU) on July 18, 2022, using the Qualtrics platform, and email invitations were sent out to a large number of contacts at state DOTs. Several reminders were sent to the survey participants to increase the response rate. The survey remained open until September 16, 2022, to allow for participation from as many states as possible. This Chapter documents the results and findings of the survey. The questionnaire used for the survey is provided in Appendix A. The questionnaire includes three main sections as follows:

Part A: Introduction/Familiarity with ML Methods

• Geographical and Organizational Context: Questions A1 and A2 ask about the location and type of the respondent’s agency, respectively.
• Departmental Unit: Question A3 inquires about the specific department within the agency where the respondent works.
• Familiarity with ML: Question A4 gauges the agency’s familiarity with ML methods and tools.
• Data Science Capacity: Question A5 checks if the agency has a Data Scientist/Engineer position or similar skill sets.
• Current Use of ML Applications: Question A6 asks if the agency currently has any ML applications deployed or in development.
• ML Methods Used: Question A7 seeks information on the specific ML methods the agency has used or is developing.

Part B: ML Applications Developed/Used In-House or Deployed as Commercial Products

• In-House ML Applications: Question B1 asks about the ML applications developed in-house that the agency is currently using.
• Commercial ML Products: Question B2 inquires about the commercial ML products the agency is currently using.
• Future ML Applications: Question B3 explores additional ML applications the agency is considering for future implementation.
• Application Areas for ML Solutions: Question B4 lists various application areas to identify where the agency has developed, implemented, or procured ML solutions.
• Satisfaction with ML Applications: Question B5 asks about the agency’s satisfaction level with the ML applications they are using.

• Detailed Information on a Widely Adopted ML Application: Questions B6-a to B6-i request detailed information about one of the most widely adopted ML applications the agency uses, including its maturity level, input data type, implementation duration, user base, implementation scale, operating costs, benefits, challenges, and lessons learned.

Part C: Ongoing and Future Development of ML Applications

• Future ML Applications: Question C1 asks about ML applications currently being explored or considered for future implementation.
• Motivation for Using ML: Question C2 seeks the agency’s main motivation for considering or using ML methods.
• Challenges in ML Adoption: Question C3 inquires about the challenges the agency anticipates in the development and adoption of ML applications.
• Vision for ML Adoption: Question C4 asks for a brief description of the agency’s vision for adopting ML methods and applications in the near future.
• Roadmap for ML Adoption: Question C5 checks if the agency has a roadmap for ML adoption and requests a brief description of it.

If the respondents indicate that their agency is using ML in question A6, they complete Part B. Otherwise, they move directly to Part C of the survey. The findings from the survey are summarized below by providing statistics and/or comments received for each one of the survey questions.

Part A: Survey Participants’ Introduction/Familiarity with ML Methods

A1. What state is your agency located in?

The research team received 43 survey responses from 29 states (Figure 6) and more than one response was received from a few large states.

A2. Which one of the following best describes your agency/organization?

Of the 43 responses received from different agencies, 42 of them identified with state DOTs, and one response was received from a County Transportation Authority as shown in Table 3.

Table 3 Responses to the survey by agency/organization type.

A3. Within your agency, please select an option that best describes the departmental unit you work for.

Participants were asked to identify the department unit they represent. Table 4 shows the broad functional representation of the units that participated in the survey.

Table 4 Responses to the survey by the primary function of the department unit.

A4. How familiar is your agency with ML methods and tools (examples include artificial neural networks, deep learning, decision trees, support vector machines, K-Nearest Neighbors (KNN), K-means, random forest, logistic regression, gradient boosting methods, reinforcement learning)?

When asked about familiarity with ML methods and tools, 23 respondents indicated that they were not very familiar with ML methods and tools (see Table 5). 14 respondents indicated that they are either very familiar or somewhat familiar with ML methods and tools. This shows that only one out of three respondents had some knowledge in this area.

Table 5 Familiarity of respondents with ML tools and methods.

A5. Does your agency have Data Scientist, Data Engineer, or Data Analyst employees? How many Data Scientists, Data Engineers, or Data Analysts does your agency employ? Please enter the number of people employed by your agency in these positions.

Respondents were also asked if their agency employed Data Scientists, Data Engineers, or Data Analysts, as shown in Table 6. The number of data scientists, data engineers, or data analysts employed by the agency ranged from 1 to 264. Additionally, less than half of respondents indicated they had positions with their agency that provided data scientist-type services or had similar skill sets. The number of people employed in these roles ranged from 1 to 10. The results indicate some disparity in the needs for skills in data science from one agency to another, depending on the state and the size of the agency.

Table 6 Data science competencies within the agency.

A5. Does your agency have any ML applications (models or tools which have been shared for use by agency staff) currently deployed and/or being developed?

19 respondents indicated that their agency had ML applications currently deployed and/or being developed as shown in Table 7. Those who responded with a ‘yes’ to this question came from 15 distinct

states out of the 29 states that responded to the survey. This shows that about half of the participating agencies have adopted or are about to adopt ML in some of their applications.

Table 7 Currently deployed or under development ML applications.

A5. What ML methods has your agency used or are in the development phase for one or more applications?

When asked about the most identified ML method used by their agencies or those that are in the development phase for one or more applications, the responses identified Artificial Neural Networks (ANN) and Deep Learning (DL) as the most common category as shown in Table 8. One respondent indicated that their agency was using/developing support vector machines (SVM).

Table 8 ML Methods adopted or being developed for applications.

Part B: ML Applications Developed/Used In-House or Deployed as Commercial Products

Respondents were asked to identify for which application areas their agency developed, implemented, or procured ML solutions.

B1 For which of the following application areas has your agency developed/implemented/procured ML solutions?

Table 9 shows that more than one-quarter of respondents identified asset management and maintenance identified transportation systems management and operations (TSMO). One respondent indicated transportation planning demand forecasting, and land use.

Table 9 Application areas with ML solutions adopted by the agency.

B2 Which in-house ML applications (including applications developed that leverage available open-source code such as pre-trained neural networks) is your agency currently using? Please indicate the applications that your organization uses the most often and indicate whether you utilize open-source code.

When asked to identify in-house ML applications being used by the agency, the respondents provided a broad range of responses as listed below.

• In-house applications are currently in the pilot phase, and the remaining applications are vendor-based/closed-source.
• ML from our video log to identify crosswalks.
• N/A for Bridge Asset Management.
• None. One project to ID wrong-way drivers is under development. DOT also had a pilot 2-3 years ago using mobile equipment and machine learning. We have a grant application ready for when a grant opportunity arises. DOT has also used ML to assess guardrail terminal type in the past.
• Not certain which applications are being evaluated.
• Predictive Travel Times.
• Project Cost Model implemented in Python using SciKit-Learn.
• Python, we are exploring SageMaker.
• Tensorflow with imagery gathered by the DOT and some outsourced imagery.
• Under development.
• Vendor-procured systems.

B3 What commercial ML products (e.g., video analytics programs for traffic volumes, algorithms for detecting incidents, image-based methods for detecting pavement or road conditions) is your agency currently using?

When asked to name commercial ML products adopted by the agency, the respondents provided the list below.

• Agency not currently using this capability but participating in a research project with a local university.
• Commercial vehicle detection, congestion mobility monitoring, intersection near-miss detection.
• DataRobot MLOps, ESRI product, Milestone VMS, Bosch Analytical Cameras.
• Developed by researchers at a university for the prediction of deterioration curves for structures.
• HQ Division of Maintenance: Video analytics for asset management (Pathweb) - this product is in regular use. HQ Division of Traffic Operations: Video analytics for traffic count (Google Cloud) - this product is in the piloting phase. HQ Environmental Analysis: Computer vision for field litter assessments (Microsoft Custom Vision) - this product was used in a previous pilot but not adopted for additional use.

• In-house development via a research project.
• Nebraska DOT used a vendor.
• No knowledge of the specific commercial ML products being considered.
• Pathway Videolog.
• Under development.
• We are looking into these.

B4 Other than the ML applications currently being used, what additional ML applications are currently being explored or are under consideration for future implementation by your agency? List as many as possible and their application areas.

When asked to name additional ML applications being explored for future implementation by the agency, the respondents provided the list below.

• Automated incident detection, transit signal priority.
• Bike/Ped Predictive Mobility Model, Safety SPF Development, Pavement Deterioration Model, Structures Improvement Costing Model, Asset Inventory using computer vision applied to 2D roadway imagery and/or LiDAR point clouds.
• Currently exploring ML for bridge deterioration models.
• None at this time. We are very early in this stage and need more pilot efforts and experience.
• Not able to respond with respect to all areas of Wisconsin DOT.
• NVIDIA traffic identifications (TensorFlow powered).
• Pavement evaluation (distress and patching identification).
• Predictive incident prediction, roadway conditions due to weather & climate change.
• Vide analytics, incident detection, pavement, and road condition detection.

B5 How satisfied is your agency with the ML applications currently in use or development? (e.g., in terms of meeting original objectives, producing implementable results, etc.)

Based on the responses received, Table 10 shows that satisfaction within respondents’ agencies for ML applications currently in use or development was high with 12 respondents reporting being somewhat satisfied and another 1 respondent being very satisfied. 1 respondent reported their agency is not at all satisfied with ML applications currently in use or development.

Table 10 Level of satisfaction with ML applications in use.

B6 Please name one of the most widely adopted ML applications that your agency currently uses most often

Respondents were asked to name one of the most widely adopted ML applications that their agency currently uses most often and answer a series of questions about that application. Names/descriptions of these widely adopted applications included:

• A Fatigue Assessment Framework for Steel Bridges using Fiber Optic Sensors and Machine Learning
• Bridge Management
• Clean California
• Custom-programmed wildlife capture software
• DataRobot
• Identifying crosswalks
• Predictive Travel Times
• Project Cost Model
• Truck Parking
• Video analytics for asset conditioning (Pathweb)

Of those who answered questions about the most widely adopted ML application, 10 respondents characterized that application as R&D (application is in the research and development phase), while 5 respondents characterized the application as Production (application is in production and has been implemented in the field) as shown in Table 11. 11 percent indicated Prototype (the application has been tested and is now a prototype).

Table 11 Level of maturity of the ML applications.

When asked about the type of input data used for the ML applications in use, nine respondents identified image data (off-line analysis), while 3 respondents indicated real-time video or real-time sensor data as shown in Table 12. 6 respondents selected “Other” for the type of input data and listed the types below.

• Databases and document libraries from previous highway construction projects
• INRIX Data
• NBI data, weather data, traffic data
• Structured/tabular data
• Tabular, geospatial
• Text

Table 12 Type of input data for ML applications.

When asked about the period the agency needed to develop and implement the ML application, the majority of respondents indicated that it took two years or less for their agency to fully implement/develop this application, with eight respondents responding 1-2 years and another 3 respondents indicating it took less than one year as shown in Table 13.

Table 13 Development/implementation period for ML application.

When asked how long the agency has been using this ML application in practice, nine respondents reported that their agency has been using the application for less than one year, while 3 respondents have been using the ML application for 3-4 years and two respondents have been using it for 1-2 years as shown in Table 14.

Table 14 Years ML application has been used in practice.

When asked about the most typical users of the ML application, 16 respondents selected those within the agency/in-house, while 2 respondents selected other state agencies. 1 respondent selected private industry and two respondents selected others as shown in Table 15.

Table 15 Typical ML application users.

Respondents were asked to indicate the implementation scale of the ML application. 7 respondents indicated the ML application is a single site. Additionally, three respondents indicated the ML application is used regionally and six respondents indicated it is used statewide as shown in Table 16.

Table 16 Implementation scale of ML application.

When asked to provide an estimate of the annual operating cost of the ML application, 10 respondents reported that the cost was less than $50,000. Very few reported the annual cost to exceed $300K as shown in Table 17.

Table 17 Estimated annual operating cost of ML application.

The survey participants were also asked to identify the main benefits of adopting ML methods for this particular application and their responses varied widely. Below is a summary of the benefits identified by the respondents.

• Automated detection of pavement patches for pavement evaluation
• Camera vendor agnostic detection
• Develop and maintain inventory
• No need for an extensive staff of data scientists
• Provide an initial inventory for crosswalks/guardrails
• Improved estimates to support more efficient decision-making
• Detection of wrong-way drivers to reduce crashes and improve safety
• Support integrated corridor management operations and predictive action plans
• Minimize the manual work involved in processing video data and take advantage of recognition capabilities for asset inventories and condition assessments
• Detection of animal movement in videos
• Wider coverage of a large network of highways
• Examples of challenges or issues faced by the organization while developing/implementing the most widely adopted ML application included:
• Gaining adoption and support
• Identifying the best method of image recognition
• In-house staff level of readiness to utilize AI/ML and ability to interpret results
• Technical issues associated with the complexities of high-accuracy classification

Part C: Ongoing and Future Development of ML Applications

Those respondents who indicated that their agency did not have any ML applications currently deployed and/or being developed were asked what ML applications are currently being explored or are under

consideration for future implementation by their agency. Examples of ML applications that are currently being explored/under consideration for future implementation include:

• DSS for ICM
• Sign identification from ROW Imagery, Pavement Marking Identification
• Smart Signals, Integrated Corridor Management, Autonomous Vehicles
• Research-based tools for analyzing data from UAS
• Tensorflow
• Vehicle classification, Ped detection

As shown in Table 18, the most common response regarding the agency’s motivation for considering/using ML methods for developing current/future ML applications was the effectiveness of ML methods compared to traditional methods. Further, 10 respondents indicated the main motivation was processing large amounts of data, and 10 respondents indicated reduction in labor costs.

Table 18 Motivation for the adoption of ML methods/solutions.

All respondents were asked what the top three challenges their agency foresees in the development and adoption of future ML applications. As shown in Table 19, the two most commonly reported challenges were a lack of AI/ML skilled workforce (27 respondents) and integrating with existing processes and systems (24 respondents). Respondents also indicated concerns with a lack of dedicated funding (12 respondents), maturity of ML technology (11 respondents), lack of other resources (9 respondents), and data availability (8 respondents).

Table 19 Current and future challenges in the development and adoption of ML applications.

Many respondents indicated that their agency does not have a vision for adopting ML methods and applications in the near future nor do they have a roadmap for ML adoption.

Summary

Below is a summary of the key findings from the survey. In the spirit of employing ML in practice for efficiency, the data collected from the surveys are fed to OpenAI’s ChatGPT, and it is then prompted to summarize the survey results in three to four paragraphs. The bulk of the text below is generated by ChatGPT-4 but a few corrections and revisions were made to improve accuracy and readability.

The survey responses provide a comprehensive overview of the current state of machine learning (ML) adoption and familiarity across 29 different state DOTs in the United States. The survey reveals a varied level of familiarity with ML methods and tools among these agencies. Over half of the respondents admit to being not very familiar with ML, indicating a gap in knowledge and application of these technologies within the sector. Despite this, there is a notable interest in developing and deploying ML applications, with 13 agencies reporting current development or deployment efforts. The employment of data scientists, engineers, or analysts in over half of the agencies suggests a growing recognition of the importance of data science competencies in leveraging ML technologies.

The survey also sheds light on the specific application areas and types of ML methods being explored or utilized by transportation agencies. Asset management and maintenance, along with transportation systems management and operations, emerge as primary areas where ML solutions are being adopted. The use of artificial neural networks and deep learning is particularly prevalent, reflecting the trend toward more sophisticated ML techniques to address complex transportation challenges. Despite the enthusiasm for ML applications, the survey indicates a cautious approach, with many applications still in the research and development or prototype stages. This is further reflected in the satisfaction levels with current ML applications, where the majority of respondents report being somewhat satisfied, suggesting room for improvement in meeting agencies’ objectives and expectations.

Looking toward the future, the survey highlights several challenges and motivations driving the adoption of ML in transportation agencies. The effectiveness of ML methods compared to traditional approaches and the potential for processing large amounts of data and reducing labor costs are significant motivators. However, the lack of an AI/ML skilled workforce and challenges integrating ML with existing processes and systems are seen as major barriers to further adoption. This indicates a critical need for skill development and organizational change to fully leverage ML technologies. Despite these challenges, the interest in exploring new ML applications, such as automated incident detection and predictive models for mobility applications, suggests a forward-looking stance among transportation agencies, albeit tempered by a realistic assessment of the hurdles to widespread ML implementation.