Data Collection Practices for Use with In-Service Performance Evaluations (2024)
Chapter: 4 Pilot Test
4 PILOT TEST
ISPE Data Collection Procedures Manual was piloted by the Maine Department of Transportation (MaineDOT), Utah Department of Transportation (UDOT), Tennessee Department of Transportation (TDOT), Iowa Department of Transportation (IowaDOT), and the Arizona Department of Transportation (ADOT). A training meeting was conducted followed by multiple individual meetings with each pilot state. The pilot states provided valuable feedback which was incorporated into the ISPE Data Collection Procedures Manual. The various activities of the pilot test are described here.
4.1 TRAINING MATERIALS AND KICK-OFF MEETING
The research team developed training materials, in the form of a PowerPoint presentation, for the data collection procedures that were developed during this project. These training materials were developed to aid in both the pilot test and ultimately in implementing the research products more broadly.
The pilot test kick-off meeting was held virtually with the five pilot states (i.e., Arizona, Iowa, Maine, Tennessee, and Utah) on April 03, 2023. The kick-off meeting covered the overall objectives of the project and the specific objectives of the pilot test. Additionally, the ISPE Data Collection Procedures Manual was introduced to the kick-off meeting attendees. A wide variety of support was provided to the states following the kick-off meeting. The available support included follow on training which could be conducted over telephone, email, virtual video conferencing or in-person on-site. Training topics suggested by the research team included:
- Compiling the ISPE dataset from multiple sources.
- Incorporating field measurements into crash response.
- Establishing a crash notification procedure.
- Step-by-step data collection training for an ISPE.
- Other topics as requested by the pilot testers.
The research team provided the pilot states with the draft ISPE Data Collection Procedures Manual the day after the kick-off meeting, on April 04, 2023.
4.2 STATE SUPPORT AND RESULTS
MaineDOT completed a table of data fields where they recorded the source data and level of ease for collecting each data element in the ISPE Data Collection Procedures Manual. Two video conferences were conducted with MaineDOT personnel. The first video conference was to clarify some of the data elements and answer some questions regarding the ISPE Data Collection Procedures Manual. The second video conference was a close out meeting where each data element, the data source, and difficulty to collect was discussed. This feedback was incorporated into the ISPE Data Collection Procedures Manual.
UDOT participated in the pilot test for the research which led to NCHRP Research Report 1010 and NCHRP Web-Only Document 332, where they conducted an ISPE of longitudinal barriers [2, 5]. UDOT’s prior experience with ISPEs provide them confidence in their data sources for both crash data and asset inventory. UDOT’s main ISPE related concern was to identify a study objective that could ensure non-biased crash data collection that also constituted a reasonable workload for staff. UDOT was interested in limiting the number of crashes to limit data collection. UDOT recognized that limiting the data by crash severity would bias the results to
high severity crashes. The research team conducted a video conference with UDOT to discuss these challenges. Ultimately, UDOT decided that limiting the study area to one or two districts in the state or filtering by roadway type would provide meaningful results while reducing data collection.
TDOT had multiple internal meetings after reviewing the ISPE Data Collection Procedures Manual which led to their feedback. Their data collection table identified the source of data for each of the ISPE data elements that are discussed in the ISPE Data Collection Procedures Manual along with the level of ease to collect each data element and comments on best practices, challenges, or problems.
IowaDOT enlisted InTrans, part of the Iowa State University (ISU), to review the ISPE Data Collection Procedures Manual and determine the level of effort and data sources required to collect the data elements defined in the Manual. InTrans has a thorough knowledge of the IowaDOT crash database and many of Iowa’s roadway asset databases. IowaDOT was able to set aside funds to support InTrans during this pilot test. IowaDOT and InTrans did not request additional training during the pilot study period. At the conclusion of the pilot test they submitted a table which summarizes the level of data collection ease and data source for each data element.
ADOT reviewed the ISPE Data Collection Procedures Manual. ADOT also participated in the pilot test for the research which led to NCHRP Research Report 1010 and NCHRP Web-Only Document 332, where they conducted an ISPE [2, 5]. ADOT’s prior experience with ISPEs provides confidence in their data sources for both crash data and asset inventory.
Table 1 provides a comparison of the difficulties, as reported by the pilot states, for collecting the identified data elements. A numeric scale has been used to describe difficulties where 1 represents the easiest elements and 6 represents the hardest elements to collect. A value of 0 represents no comment. Although there is some variation between the states, the data elements categorized as routine are generally easy for the jurisdictions to collect. The data elements categorized as investigative data elements are more difficult than the routine elements but generally still possible to collect. The data elements categorized as reconstruction or asset management tend to be more difficult for these agencies to collect.
The pilot states reported obstacles to utilizing some identified available data sources. The pilot states observed that the available data is often maintained by different divisions within the pilot states’ agencies or by other agencies all together (i.e., law enforcement). One noted obstacle was data mapping difficulties when trying to link multiple databases, especially when there is not a common data field between the databases. The different organizational structures, work force availability, and departmental priorities can create challenges in timely delivery of the ISPE data. Overcoming these obstacles necessitates coordination with these other groups early in the data collection process to educate the other groups about the importance of ISPEs. Furthermore, solving database communication difficulties can often be accomplished by identifying a linking data element and gathering that element when conducting an ISPE. Twelve unique data sources were identified by the pilot states, including:
- Crash Reconstruction
- Field Inspection
- Insurance Recovery Records
- Maintenance Records
- Contract Documents
- Construction Records
- Work Request
- Hardware Inspection Reports
- Roadway Logs
- Asset Management / Asset Inventories
- Photographic Data
- Video Data
Using these data sources necessitates an understanding of the element available and working with the people who maintain the data. An understanding of each database and the people who maintain the database can be helpful. Recognizing how their time is allocated is also helpful. Finally, it may be helpful, when appropriate, to provide additional training to improve the amount of data routinely available for ISPEs.
Table 1. Comparing the level of difficulty to collect data elements.
| Data Element | Nomenclature | State A | State B | State C |
|---|---|---|---|---|
| Safety Feature Under Evaluation | SFUE | 1 | 1 | 3 |
| Crash Record Number | CRN | 1 | 1 | 1 |
| Crash Date | CRASH_DATE | 1 | 1 | 1 |
| Crash Location | CRASH_LOC | 1 | 1 | 1 |
| Number of Units | TOTAL_UNITS | 1 | 1 | 1 |
| Maximum Crash Severity | MAX_SEV | 1 | 1 | 1 |
| Vehicle Type | VEH_TYPE | 1 | 1 | 1 |
| Posted Speed Limit | SPEED_LIMIT | 1 | 1 | 1 |
| Harmful Event Post Impact with Safety Feature Under Evaluation | PostHE | 1 | 1 | 1 |
| Most Harmful Event Crash with the Safety Feature Under Evaluation | MHE | 1 | 1 | 1 |
| First Harmful Event Crash with the Safety Feature Under Evaluation | FHE | 1 | 1 | 1 |
| Any Harmful Event Crash with the Safety Feature Under Evaluation | AHE | 1 | 1 | 1 |
| First and Only Harmful Event Crash with the Safety Feature Under Evaluation | FOHE | 1 | 1 | 1 |
| Roadway Surface Condition | RSUR | 1 | 1 | 1 |
| Safety Feature Breach | BREACH | 4 | 5 | 5 |
| Predictable Breakaway | BREAK | 4 | 5 | 5 |
| Controlled Penetration, Redirection, or Stop | PRS | 4 | 5 | 5 |
| Safety Feature Penetration into Occupant Compartment | PEN | 4 | 5 | 5 |
| Initial Contact Point | ICP | 5 | 5 | 3 |
| Sub Feature Name | NAME | 3 | 5 | 5 |
| Safety Feature Offset | OFFSET | 2 | 3 | 5 |
| Safety Feature Height | HEIGHT | 3 | 3 | 0 |
| Curbing | CURB | 2 | 3 | 3 |
| Curb Type | CTYPE | 2 | 3 | 6 |
| Curb to Safety Feature Distance | CDIST | 3 | 3 | 5 |
| Tapered Edge | EDGE_TAP | 2 | 3 | 0 |
| Ground Condition | GRNDCOND | 1 | 3 | 6 |
| Data Element | Nomenclature | State A | State B | State C |
|---|---|---|---|---|
| Initial Contact Point with Safety Feature Under Evaluation | IP_SFUE | 5 | 0 | 5 |
| Impact Speed | ISPEED | 5 | 0 | 0 |
| Velocity Angle | VANGLE | 5 | 0 | 6 |
| Heading Angle | HANGLE | 3 | 0 | 6 |
| Yaw | YAW | 3 | 0 | 5 |
| Safety Feature ID | SFID | 1 | 1 | 5 |
| Safety Feature Location Beginning and End | SFLOC_B & SFLOC_E | 1 | 5 | 5 |
| Installation Inspected and Documented | INSTALL | 3 | 3 | 6 |
| Maintenance Inspection | MAINT | 1 | 3 | 5 |
| Installation Date | I_DATE | 0 | 3 | 6 |
| Installation Cost | I_COST | 2 | 5 | 6 |
| Repair History | R_HIS | 5 | 3 | 6 |
| Repair Type | R_TYPE | 0 | 5 | 6 |
| Repair Cost | R_COST | 0 | 5 | 6 |
