CHAPTER 5. CHANGE ORDER ANALYSIS

INTRODUCTION

This chapter summarizes the results of an analysis of sample change order data the research team received from state DOTs. As part of the practitioner survey (see Appendix A), respondents were asked whether their agency could provide a copy of change order and claim databases to the research team. For DOTs that responded on the affirmative, the research team followed up with emails and formal requests through official channels, including open-record requests.

Table 42 shows a list of change order and claim data that was available for the analysis. The table also includes entries for data the research team had received as part of previous research efforts going back to the late 2000s. Per NCHRP’s request, DOTs are identified by case numbers (not by name) to anonymize the discussion and results. Analyzing data from all 11 cases in detail was not feasible, so the research team focused on six cases (1, 2, 5, 6, 8, and 9), which covered a wide range of change order record practices and styles.

Table 42. Change Order and Claim Data Received from State DOTs.

¹ Data the research team received as part of previous research efforts going back to the late 2000s.

METHODOLOGY TO CLASSIFY UTILITY-RELATED CHANGE ORDERS

The research team classified individual change order records as one of the following (regardless of what the reason code indicated):

• UR: UR records were those for which the research team had a high-level of confidence the change order was UR.
• NUR: NUR records were those for which the research team had a high-level of confidence the change order was NUR.
• Utility-related suspected (URS): URS records were those for which the research team suspected the change order was UR, but the change order description was not sufficiently clear.
• Non-utility-related suspected (NURS): NURS records were those for which the research team suspected the change order was NUR, but the change order description was not sufficiently clear.

The research team used three different methodologies to classify change orders:

• Use commonly used UR terms.
• Review the description and justification columns to assess whether the change order was UR, NUR, URS, or NURS. This methodology also included classifying UR change orders according to a list of nine potential disaggregated reasons.
• Use AI models to detect trends and patterns that could indicate a change order was UR or NUR. Chapter 6 includes a detailed description of the methodology and results of this analysis. After using the AI models, the research team reviewed individual records to validate or, if necessary, change the label the AI tools predicted. The research team also used the AI models to augment the list of commonly used UR terms. In practice, the research team used the AI-generated list of commonly used UR terms to filter change orders and accelerate the review process. In turn, the research team used the result of the manual review of individual change order description and justification columns to augment the list of terms.

Commonly Used UR Terms

The research team prepared a dictionary of commonly used one-word and two-word UR terms based on the thousands of records from Cases 1, 2, 5, 6, 8, and 9 (Table 43). The dictionary of one-word UR terms includes 60 entries. The dictionary of two-word UR terms includes 148 entries. The research team also considered three-word UR terms, but the predictive power of these terms was low.

Table 43. Commonly Used One-Word and Two-Word UR Terms.

¹ HDPE: High-density polyethylene

The research team used several metrics to analyze change order records using the commonly used terms. One of the metrics was the number of times a change order description mentioned UR terms. Another metric was the number of change orders that mentioned UR terms, regardless of the number of times a change order mentioned a UR term. At first, it appeared the first metric could be a more reliable indicator of the usage of individual UR terms because it might suggest how important UR terms were for the DOT officials completing change order descriptions. In the end, the second metric, which was simpler to conceptualize and calculate, was just as effective in measuring the usage of UR terms. To normalize the metric, the research team defined relative usage of UR terms as follows:

$Relative=\frac{NumberofURchangeordersthatmentionedtheterm}{Totalnumberofchangeordersthatmentionedtheterm}$

Relative usage is an indicator of the predictive power of a term. As the relative usage increases, so does the predictive power of the term. For example, if the term “sewer” has a 100 percent relative usage, what it means is that a thorough review of all change orders that mentioned the term “sewer” were indeed UR that only UR change orders included that term.

Disaggregated Reasons for UR Change Orders

The research team used the results of the practitioner survey, as described in Chapter 3, to prepare a list of potential reasons for UR change orders. This list was more disaggregated than what DOTs normally use (e.g., DSCs, utility delays, or errors in plans and specifications). Table 44 shows the list of nine disaggregated reasons, along with a corresponding description.

As a reference, 23 CFR 635.109 has three standardized changed condition clauses in highway construction projects (unless prohibited by state law or if a DOT has developed or implemented one or more of those clauses):

• DSCs.
• Suspensions of work ordered by the engineer.
• Significant changes in the character of work.

One of the disadvantages of labeling a change order as DSC is that the risk of missing or masking the actual reason behind a change order increases considerably. If a more disaggregated list of reasons is not available, a change order might be labeled as DSC even though the cause was, for instance, that information about existing utilities was inadequate, a utility owner relocated its facility to the wrong place, or the PS&E documents had errors. As the research team reviewed individual change order records, an important focus was to find the actual reason behind the change order, regardless of whether the change order was originally labeled as DSC.

Table 44. Disaggregated List of Reasons for UR Change Orders.

CASE 1

The change order database included 1,364 records from 2007 to 2022. It included contract number, description, change order number, approval date, and amount. According to state officials, the 1,364 records correspond to change orders for which the primary cause was a utility issue. The change order database had a brief description column (on average, 4 words per change order, with a range of 1–36 and a standard deviation of 3). However, the DOT often uses acronyms in the description column. Spelling out the acronyms would have the effect of increasing the average number of words per change order.

The DOT uses four-letter codes to classify change orders. The first character corresponds to the change order type and can be one of eight codes (e.g., A for contract or supplemental work or B for acceleration). The second character corresponds to the specification or asset affected by the change order and can be one of 25 codes (e.g., C for different site conditions or L for utility). The third and four characters correspond to the source of the document that created the need for the change order and can be one of 43 codes (e.g., PV for utility plans or SB for standard specifications). The database the research team received did not include the four-letter codes, but it is reasonable to assume all or most records had L in the second code or PV in the third and fourth codes.

For the analysis, the research team used the description column to examine each record in the database. Of the 1,364 records in the database, the research team had to assume that all the records were UR records because there was not a practical way to prove the opposite. Using the change order reasons in the database, the research team prepared Table 45, which shows the number and percentage of UR change orders for each disaggregated change order reason.

Table 45. Case 1: Reasons for UR Change Orders.

Change orders classified as other were challenging because the description column did not provide much valuable information to assess what the actual UR reason for the change order was. Examples of those descriptions were as follows:

• Additional funds.
• Delineators.
• Irrigation controller various locations.
• Place additional inductive loops.
• Water connection and capacity fees.

Labeling records as inaccurate or incomplete data about existing or relocated utility facilities was also challenging because the change order description did not normally include wording confirming this reason. As a strategy to deal with this issue, the research team assumed that if a utility facility was a “high-priority” facility, the DOT probably commissioned a test hole during the design phase to confirm the X-Y-Z coordinates of the facility. High-priority facilities include:

• Natural gas pipelines larger than 15 cm (6 in) in diameter or with an operating pressure greater than 414 kPa (60 psi).
• Petroleum pipelines.
• Pressurized sanitary sewer pipelines.
• High-voltage electric supply lines, conductors, or cables of at least 60 kV.
• Pipelines transporting hazardous materials.

Other utility facilities are not high-priority facilities and, therefore, the probability of having a test hole to confirm their coordinate would have been low. The DOT normally does not use quality level B (QLB) utility investigations during the design phase. This means that for low-priority facilities, the level of uncertainty regarding their location is high (or at least higher compared to the level of uncertainty associated with high-priority facilities). Consequently, the research team assumed that change orders that involved low-priority facilities (unless the change order already pointed to another reason) would be labeled as inaccurate or incomplete data about existing or relocated utility facilities.

The research team also conducted an analysis of the frequency of single-word terms and two-word terms. Because the research team did not receive a complete database of change orders, it was not possible to calculate the relative usage of UR terms. Table 46 shows the 15 most frequently used one-word terms and the 15 most frequently used two-word terms.

Table 46. Case1: Most Frequent UR Change Order Terms.

CASE 2

The change order database included 4,225 records from 2010 to 2022. The change order database had a brief description column (on average, 12 words per change order, with a range of 1–36 and

a standard deviation of 3). The DOT uses change order reasons such as DSCs, utility delays, plan or specification errors, plan or specification differing interpretations, time or schedule, right-of-way, and urgent need. The data includes several other change order reasons (e.g., added work, added funds available, alterations to contract, change basis of payment, conformity to the contract, and contract issues) as well as variations of the same reasons (e.g., utility delay, utility delays, and delays).

For the analysis, the research team used both the description and change order reason columns to examine each record in the database. Of the 4,225 records in the database, the result was 197 (4.7 percent) UR records, 3,044 (72.0 percent) NUR records, 81 (1.9 percent) URS records, and 903 (21.4 percent) NURS records. The number of URS and NURS records was high because the description column was too short to enable a reliable classification of those records as UR or NUR.

Using the change order reasons in the database, the research team prepared Table 47, which shows the number and percentage of UR change orders. Because of variations in change order reason naming conventions and spelling, the research team grouped the results into five meaningful categories of change order reasons.

Table 47. Case 2: UR Change Order Classification.

Using the list of disaggregated change order reasons in Table 44, the research team also prepared Table 48. Although the numbers and percentages changed, using disaggregated reasons did not really help to uncover more information compared to what the DOT’s list of change order reasons already provided. The small number of words in the change order descriptions (four words on average) precluded a more detailed analysis.

Table 48. Case 2: Disaggregated Reasons for UR Change Orders.

The research team also conducted an analysis of the relative usage of single-word terms and two-word terms. Table 49 shows the one-word terms that had the highest relative usage. Two-word terms were not particularly effective because of the small number of words in the change order description column. Some two-word terms were mentioned a few times, including fiber optic, line relocation, water relocation, utility relocation, and water line. Utility delay was the most common UR two-word term (although this term was already one of the standard change order reasons the DOT uses).

Table 49. Case 2: Strongest UR Change Orders Predictors.

CASE 5

The change order database included 11,659 records from 2007 to 2021. It includes contract number, change order number, dollar amount, description, and remarks. The DOT does not use reason codes. The change order database had detailed description and remarks columns

(combined on average, 130 words per change order, with a range of 4–1,825 and a standard deviation of 112). The remarks column usually included the list of modified items and the reason for that change.

For the analysis, the research team used both the description and remarks columns to examine each record in the database. Of the 11,659 records in the database, the result was 460 (3.9 percent) UR records, 10,199 (87.5 percent) NUR records, 21 (0.2 percent) URS records, and 979 (8.4 percent) NURS records. The percentage of records classified as NURS was high compared to other states. Most of the records grouped as NURS included a list of items with final quantities (i.e., quantities that were measured during construction), which made it difficult to decide whether the change order was UR.

Using the list of disaggregated change order reasons in Table 44, the research team prepared Table 50, which shows the number and percentage of UR change orders for each disaggregated change order reason.

Table 50. Case 5: Reasons for UR Change Orders.

In most cases, the description and remarks columns were sufficient to decide whether a change order was UR. However, several records lacked clear information to enable a reliable classification. In other cases, the change order did not mention the reason, but listed the modified items, which enabled the research team to make a reasonable assumption about the proper classification for those change orders. Several change orders showed a list of items adjusted to final quantities in the description, but the description did not include more information. For those change orders, the research team decided to classify them as errors and omissions in PS&E, but it is possible that in some cases the actual reason might have been different.

In an effort to speed up the classification process, the research team filtered change orders using the list of one-word terms and two-word terms (Table 43). However, this approach was not particularly effective because change orders often included different terms to name the same structure or feature.

The research team also conducted an analysis of the relative usage of single-word terms and two-word terms. Table 51 shows the one-word terms and two-word terms that had the highest relative usage.

Table 51. Case 5: Strongest UR Change Orders Predictors.

CASE 6

The research team received a supplemental agreement database, which included 19,912 records, and a claim database, which included 9,151 records (of which 6,613 records corresponded to approved claims). Both databases included records from 2007 to 2022. The DOT uses supplemental agreements to amend the contract in situations where there are changes such as work requirements, unit prices, additional work items, or contract times. The DOT uses claims in situations where the contractor requests additional compensation or a time extension. Contractors can submit claims either during the life of the contract or after the construction ends. A review of records from both databases revealed that most records corresponded to situations that other DOTs would handle by using change orders. For completeness, the research team processed each database separately but then combined the results to enable an aggregated comparison with other cases.

The supplemental agreement database had a high-level of detail in the description and justification columns (combined on average, 199 words per supplemental agreement, although the range was 3–4,937 words and the standard deviation was 171). This database did not include change order reason codes. The claim database also had detailed descriptions, although the number of words per claim was lower than the number of words for supplemental agreements (43 words per claim on average, range of 3–337, and standard deviation of 23).

For the supplemental agreement database, the research team used both the description and justification columns to examine each record in the database. Of the 19,912 records in the database, the result was 2,624 (13.2 percent) UR records, 17,256 (86.7 percent) NUR records, 27 (0.1 percent) URS records, and 0 (0.0 percent) NURS records. The number of URS and NURS records was low (and the percentage was lower than for Case 2) because the description column was sufficiently long to enable a reliable classification of change order records as UR or NUR.

For the approved claim database, the research team used both the claim description and the issue description columns. Of the 6,613 records in the database, the result was 884 (13.4 percent) UR records, 5,729 (86.6 percent) NUR records, 0 (0.0 percent) URS records, and 0 (0.0 percent) NURS records.

Using the list of disaggregated change order reasons in Table 44, the research team prepared Table 52, which shows the number and percentage of UR supplemental agreements, Table 53, which shows the number and percentage of UR claims, and Table 54, which shows the combined results, for each disaggregated change order reason.

Table 52. Case 6: Reasons for UR Supplemental Agreements.

Table 53. Case 6: Reasons for UR Claims.

Table 54. Case 6: Reasons for Combined UR Supplemental Agreements and Claims.

In most cases, the description and justification columns were adequate to decide whether a supplemental agreement or claim was UR. Nevertheless, the process was tedious and time consuming. Despite using single-word terms and two-word terms to accelerate the process, it was necessary to read the description and justification columns carefully in search of the one sentence in the middle of the description and/or justification entries that included the critical piece of information.

The description and/or justification columns for some supplemental agreements was particularly challenging, which needed further discussions within the research team. A typical example of this type of situation was the following:

[The] waterline is not in conflict with construction but was disturbed by nearby pavement removal operations. The City requested the relocation of a new waterline and encasement pipe to safeguard against future maintenance issues.

In this case, the waterline was not in direct conflict with the highway construction, but it was affected by pavement removal operations nearby. At first, it appeared that the pavement removal occurred outside of the construction area or the project limits (maybe as part of a local project). The city’s request to relocate the existing waterline and encasement pipe would mean the supplemental agreement would be classified as changes initiated by project owner, contractor, or utility owner. However, it is also possible the pavement removal occurred elsewhere within the project limits, in which case the waterline would have been in conflict with the project (although not in conflict at the specific location where the city requested to relocate the waterline and encasement pipe). If so, a more appropriate reason would have been inaccurate or incomplete data about existing or relocated utility facilities.

The research team also conducted an analysis of the relative usage of single-word terms and two-word terms. Table 55 shows the one-word terms and two-word terms that had the highest relative usage.

Table 55. Case 6: Strongest UR Supplemental Agreement Predictors.

CASE 8

The change order database included 237 records from 2011 to 2021. It included item description, approval date, dollar amount, and explanation. The 237 records received correspond to change orders that included references to utility delays in the description or explanation columns (i.e., records having one of the following keywords: “*UTILITY DELAY*,” “*Utility Delay*,” or “*utility delay*”). The description and explanation columns included a detailed description of the change orders (combined on average, 247 words per change order, although the range was 29–1,203 words and the standard deviation was 30).

For the analysis, the research team used the item description and explanation columns to evaluate each record. Using the list of disaggregated change order reasons in Table 44, the research team prepared Table 56, which shows the number and percentage of UR change orders for each disaggregated change order reason.

Table 56. Case 8: Reasons for UR Change Orders.

It is not surprising that the highest number of records was associated with delays getting utility owners to schedule utility relocations. The reason is that all records received from the DOT were already filtered since they included the term “utility delay.” In most cases, the explanation column included a detailed description of activities, as well as the party responsible for decreasing the impact of the delay. Many of the records described other necessary activities because the delay triggered additional winter-related delays or stoppages.

Although all the records had a utility delay filter, a substantial number of records had content in the description column that pointed to a different reason, suggesting in those cases that utility delay was an effect, not a cause. An example was a change order in which the contractor found that the vertical clearance of some existing aerial electrical cables would be insufficient during the construction of a bridge. The contractor could not proceed until the issue was resolved. In the end, it became necessary for the utility owner to re-route electrical circuits and deenergize the line in question. Although there was a delay, the issue that caused the delay during construction was not identified during the design phase.

In other cases, utility delay was not even an effect. An example was a change order in which the contractor found a utility facility that was not mapped during design, causing a redesign of parts of the drainage system. In this case, it was not necessary to relocate the utility facility.

The research team also conducted an analysis of the frequency of single-word terms and two-word terms. Because the research team did not receive a complete database of change orders, it was not possible to calculate the usage of UR terms. Table 57 shows the 15 most frequently used one-word terms and the 15 most frequently used two-word terms.

Table 57. Case 8: Most Frequent UR Change Order Terms.

CASE 9

The change order database included 104,540 records from 2000 to 2023. It included project number, change order number, approval date, change order amount, reason codes, description, and remarks. Users can assign up to three reason codes per change order (primary, secondary, and tertiary) depending on the number of changes included in the change order. The change order database had a high-level of detail in the description and remarks columns (combined on average, 242 words per change order, although the range was 1–5,123 words and the standard deviation was 181).

For the analysis, the research team used the reason codes and the description and remarks columns. The procedure to classify records as UR or NUR consisted of two phases. First, the research team extracted 2,790 change orders with right-of-way or UR reason codes and then added change orders that had reason codes associated with incorrect PS&E, third party requests for additional work, and additional work desired by the DOT. The reason for adding these change orders was that a preliminary analysis showed a considerable number of them were UR. The result of the first phase was a subset of 19,100 change orders. For each record, the research team reviewed the description and remarks columns to decide whether the change order was UR or NUR. At the end of this first phase, 3,945 records were classified as UR.

Second, the research team used AI models to accelerate the review of the remaining 85,440 change order records in the database. The process to review every single record manually in the 19,100-record subset was tedious and time consuming, and completing the classification of the remaining records by hand would have taken several months. Chapter 6 describes the process to select, train, and validate the AI models. This pre-classification enabled the research team to

review the description and remarks columns faster. For the 85,440 change orders, the result after the review was 3,512 UR records and 81,928 NUR records.

In total, of the 104,540 records in the database, the result was 7,457 (7.1 percent) UR records and 97,083 (92.9 percent) NUR records. The amount of information was sufficient to classify the change orders, which made it unnecessary to use the URS or NURS labels.

Using the list of disaggregated change order reasons in Table 44, the research team prepared Table 58, which shows the number and percentage of UR change orders for each disaggregated change order reason. Table 58 shows results for 6,037 of the 7,457 UR change orders.

Table 58. Case 9: Reasons for UR Change Orders.

The description and remarks columns were usually sufficient to classify change orders. However, some change orders included just a short sentence without any further explanation. In these cases, the research team used the reason codes (and their definition) to classify the change order. If this information did not provide enough context, the research team classified the change order as other.

The research team also conducted an analysis of the relative usage of single-word terms and two-word terms. Table 59 shows the one-word terms and two-word terms that had the highest relative usage.

Table 59. Case 9: Strongest UR Change Orders Predictors.

GENERAL TRENDS

The total number of UR change orders for the six cases was 11,803. For each case, Table 60 shows the percentage of UR change orders per disaggregated change order reason. The table also shows the overall average for all cases.

Table 60. Percentage of UR Change Orders per Disaggregated Change Order Reason.

¹ Average percentage was calculated using the total number of UR change orders for each disaggregated change order with respect to the overall total of UR change orders.

The results in Table 60 point to two major groups of reasons behind UR change orders:

• Reasons behind a substantial number of UR change orders (79 percent in total):
  • Errors and omissions in PS&E (33 percent).
  • Inaccurate or incomplete data about existing or relocated utility facilities (23 percent).
  • Changes initiated by project owner, contractor, or utility owner (12 percent).
  • Delays getting utility owners to schedule utility relocations (11 percent).
• Reasons behind a small number of UR change orders (21 percent in total):
  • DSCs (4 percent).
  • Difficult or inadequate constructability of highway work or utility relocation (4 percent).
  • Inaccurate or deficient utility relocation work (2 percent).
  • Delays acquiring or clearing right-of-way or utility relocation sites (2 percent).
  • Other (9 percent).

This differentiation has significant ramifications for the identification of potential strategies for implementation. First, except for utility relocation delays caused by utility owners (12 percent of UR change orders), most reasons that cause a substantial number of UR change orders are reasons that a DOT can control, specifically, errors and omissions in PS&E (33 percent) and inaccurate or incomplete data about utility facilities (23 percent). These two reasons account for 56 percent of UR change orders.

Second, most reasons behind a substantial number of UR change orders are reasons that a DOT could address prior to letting, which highlights the importance of conducting utility investigations and identifying and resolving utility conflicts during the preliminary design and design phases. Overall, the results in Table 60 show that pursuing these two strategies systematically could have a positive impact on 60–80 percent of UR change orders.

As expected, for DOTs where the change order description was sufficient (i.e., Cases 1, 5, 6, 8, and 9), the number of UR change orders the research team classified as DSCs was low. For those states, the research team could determine the actual reason behind the change order (even if the DOT had originally classified the change order as a DSC). This result is significant because it could point to many cases in which a change order might be classified as a DSC for convenience or because the official in charge did not have more meaningful categories to choose from, but the actual reason was completely different.

Other reasons listed in Table 60 also had a small number of UR change orders. One of those reasons was difficult or inadequate constructability of highway work or utility relocation. It is likely that one of the main reasons is that the number of complex utility relocations is low in most projects and that highway contractors are often able to predict complex situations while preparing the bid for the project. Likewise, the number of UR change orders attributed to delays acquiring or clearing right-of-way or utility relocation sites was low. Reasons include that DOTs often acquire all the necessary parcels before the project is let and that highway contractors know about the pending parcels while preparing the bid and plan their sequence of work accordingly.

The number of UR change orders under the category of inaccurate or deficient utility relocation work was low. Most change orders under this category were the result of damage the utility contractor caused to new or existing highway infrastructure or situations in which the utility contractor placed a utility facility at a location different from what the approved plans showed. It is worth noting that change orders do not reflect the case where a utility facility was placed incorrectly but the resulting impact did not translate into a change order.

The research team also examined yearly variations in the number and percentage of UR change orders. As expected, trends varied from state to state. Unfortunately, it was not possible to draw meaningful conclusions from these observations. In some cases, the data the research team received included estimated project costs, but not letting dates. In addition, critical relevant information was not available to the research team, such as potential changes in change order recording practices or potential changes in utility process management practices.

Readers should be aware of caveats associated with individual databases, which makes the comparison among databases more challenging. For example:

• Case 1: The 1,364 records correspond to change orders for which the primary cause was a utility issue. The database did not include a change order code.
• Case 2: The change order database contained 4,225 records, including UR and NUR records. However, change order descriptions were too short to enable a reliable classification.
• Case 5: The change order database contained 11,659 records that included short descriptions and long remarks. However, the remarks column included mainly a list of modified items.
• Case 6: The change order database contained 19,912 records with a high-level of detail in the description and justification columns. This database did not include change order reason codes.
• Case 8: The change order database contained 237 records that included references to utility delays in the description or explanation columns.
• Case 9: The change order database contained 104,540 records with highly detailed descriptions and reason codes. Change orders included up to three reason codes.