Holistic framework for highway construction cost index development based on inconsistent pay items

A construction cost index (CCI) measures the price changes of construction items over time. It allows owner agencies and contractors to monitor construction market fluctuations so that they can more accurately estimate construction costs and project long-term funding needs. However, various CCI calculation methods in the highway construction industry are limited in two respects: (1) inability to account for inconsistent pay items, primarily caused by changes in pay item catalogs, and (2) insufficiency in statistically cleaning pay item data, e.g., unbalanced bid prices. For this reason, the resulting highway construction cost index (HCCI) cannot accurately reflect the construction market. To address such limitations, this research develops a three-step holistic framework for automated HCCI development using inconsistent pay items. The three-step methodology encompasses (1) data cleaning, where a text analysis algorithm and outlier detection algorithms are developed to clean inconsistent pay items and unbalanced bid prices; (2) pay item sampling, which helps to select and edit pay items through various statistical analysis; and (3) HCCI calculation, where the chained Fisher index formula is applied to calculate HCCIs at the state level, as well as indexes for specific regions and item categories. A prototype application is developed to generate quarterly and annual HCCIs automatically using the Python programming language. Ten-year data, i.e., 251,033 records, were used to verify and validate the prototyped system. The resulting HCCI and sub-HCCIs provide reliable insights into construction market conditions with high granularity. This research contributes to the body of knowledge by offering a holistic framework for automated HCCI development that leverages the text analysis algorithm for accounting for inconsistent pay items and two-stage outlier treatments for cleaning pay item data.