Server-customer interaction tracker: Computer vision-based system to estimate dirt-loading cycles

Real-time monitoring of heavy equipment can help practitioners improve machine-intensive and cyclic earthmoving operations. It can also provide reliable data for future planning. Surface earthmoving job sites are among the best candidates for vision-based systems due to relatively clear sightlines and recognizable equipment. Several cutting-edge computer vision algorithms are integrated with spatiotemporal information, and background knowledge to develop a framework, called server-customer interaction tracker (SCIT), which recognizes and measures the dirt loading cycles. The SCIT system detects dirt loading plants, including excavator and dump trucks, tracks them, and then uses captured spatiotemporal data to recognize loading cycles. A novel hybrid tracking algorithm is developed for the SCIT system to track dump trucks under visually noisy conditions of loading zones. The developed framework was evaluated using videos taken under various conditions. The SCIT system with novel hybrid tracking engine demonstrated reliable performance as the comparison of the machine-generated and ground truth data showed high accuracy.