CHAPTER 4

Pre-Fusion Considerations

Before officially starting the data fusion process, there are seven key items to consider or address:

1. Evaluating the expected ROI
2. Understanding the impacts to legacy systems
3. Obtaining the rights to datasets
4. Ensuring derivative products are non-sensitive and privacy-preserving
5. Choosing private-sector datasets that will be persistent
6. Ensuring the data have enough coverage of high enough quality to support a particular use case
7. Acknowledging that algorithms are not always transferable

4.1 Expected ROI

Some forms of data fusion will be low-cost, while others will be complex and potentially expensive. For some use cases, the results may be well worth the added cost. For other use cases, the incremental improvements (or even risk of no operational improvement) may not be worth the expense. Executives need to keep in mind that the expense of data fusion is not “one-and-done.” There are ongoing costs associated with maintaining the system and/or modifying the system when changes to data or other systems occur.

4.2 Impacts to Legacy Systems

Agencies are likely using both point-sensor data and probe data for other purposes already. They may be integrated into ATMS or 511 platforms, used to derive travel times for dynamic message sign (DMS) messages, used to detect the back of a queue around a dangerous corner, or even used for real-time enforcement activities. Agencies will want to ensure that the way in which the fusion team interacts with these data is non-damaging to these or other existing applications. For example, an old database server may have been just barely sized for a particular application, and any additional load on that server may introduce latency or even prevent other applications from working at all. Last, it is critical that the original data are preserved and that the fusion process does not accidentally modify or change the source data in their original location. Doing so would impact other systems currently relying on the data as is.

4.3 Obtaining the Rights to Datasets

The framework assumes that the entity responsible for implementing the fusion has already procured (or has access to) enough data to make fusion worthwhile. This document does not provide specific guidance on how to procure data. If agencies already own a point sensor or probe-based

Many believe that the inevitable march toward more and more location data may not actually be inevitable. Data may not continue to grow in the predicted trajectory. Data providers disappear frequently, often at the mercy of upstream suppliers and constant changes in relationships. The landscape is constantly shifting. What is reality today may not be reality tomorrow. Thus, validations of certain data provider products (e.g., volumes, O-Ds) that are sensitive to losses or additions to underlying base data become invalid and potentially worthless as soon as the underlying data suppliers change. For this reason, agencies should attempt to discover these changes and ask their suppliers to be open about changes. For example, TETC is trying to tackle some of these validation challenges and is working on regular validations of newer, more sensitive data products. More information on this topic may be found in Section 4.6.

Unfortunately, there have been many disruptions to the marketplace. For example, one company went into bankruptcy and other data providers have dropped data products after brief experimentation phases. Those who had invested heavily in the company that went bankrupt (both agencies and other private-sector companies) were left with products and services that no longer worked and may never work again—at least not as they once did.

Agencies should strive to work with companies that (1) have longevity in their products and their staff—which is a sign of a strong company, (2) that rely on more than one data source so that they can survive minor disruptions, and (3) that are not showing signs of potential financial or other troubles.

4.6 Ensuring Adequate Coverage and Quality

The quality and coverage of speed data has been well documented in both urban and rural environments. TETC conducts regular probe speed data quality checks on a wide variety of road classes and environments. Past data validation reports can be found here: https://tetcoalition.org/projects/transportation-data-marketplace/.

TETC is now working with a team of academics at the CATT Laboratory and a panel of state transportation experts to validate other probe-based datasets such as O-Ds, waypoints, volumes, and events. While no reports on validation have been published at the time this report was written, readers should check the TETC website for updates on these efforts.

It is currently assumed that as time progresses, more CVs will be on the road, which will ultimately mean that vendor data penetration rates will also rise. While this is the desired outcome for agencies, the reality is different: data brokers often pay the cost of pulling data directly from vehicles themselves. As more vehicles become available for data access, it is not always in the best interest of the company to download that data—especially if price lists are fixed. There may also be limits that OEMs place on the amount of vehicles available in their pools at any given time. For example, even if there are 10 million CVs on the road, the OEMs or data brokers may only provide access to a rotating pool of those 10 million vehicles so as not to track any one for too long and better ensure privacy protection.

When coverage (or penetration rates) is provided for O-D, waypoint, or event-style probe data, vendors will make claims about their ability to cover the population of a region. A typical claim may state, “our data cover 35 percent of the population” or “our data cover 5 percent of daily trips.” It is important to ask each company to define how they calculate their coverage percentages, and how they define a trip, as those with lower percentages may have more data. Examples of how these numbers can be computed are shown in Table 3.

This is not to say that one method is better (or more accurate) than another. It is mainly important that the vendor be transparent in its claims and that the purchaser be aware of the implications.

Table 3. Two examples of how vendors may classify their coverage.

Some companies are pivoting away from coverage definitions based on percentages and penetration rates for some use cases, as it may distract from the value of available data. For example, for a signalized intersection, it may be less important that a certain percentage of population is represented, but more important that there is a minimum number of samples available to extrapolate necessary metrics. These companies are now trying to state the statistical significance of their coverage rather than exact percentages (which can be difficult to quantify). Whether this clarifies the situation or is another strategy to muddy the waters is yet to be determined.

4.7 Algorithms Are Not Always Transferable

While agencies are encouraged to share fusion algorithms with one another to avoid duplication of effort and to build off each other’s knowledge, it is important to understand that specific algorithms that take into account the uniqueness of one geography, data availability, etc. may not be directly transferable to another agency with different characteristics.

For example, the National Renewable Energy Laboratory (NREL) presented a study at the 2021 TRB Annual Meeting where they tested the transferability of machine learning algorithms between states (Sakhare, Desai, et al. 2022), to see if the machine learning models created and trained against data from one state would be transferable to the other states and vice versa. The three states were Colorado, North Carolina, and Pennsylvania, and the hope was that one model could essentially be copied and pasted to other states and still expect quality results. The data used in the models included count data from point sensors, probe-based data, and weather data. The research displayed in Table 4 shows that some models worked better than others at overcoming differences between the state results. Also, states with similar geographic and weather conditions were more likely to have better results when trying to use a model developed in another state.