Multiimaging sensor data fusion-based enhancement for 3D workspace representation for remote machine operation

In incompletely characterized environments such as construction sites, remote machine operation is the preferred - and sometimes the only - safe and efficient solution for the operation of construction machines. When it comes to the operation of remote-controlled construction machines, a human - machine interface is needed so that even in the case of an unstructured environment (such as a construction site), the operator can interact with the machine in a safe and efficient manner. The human - machine interface needs to have the capability of realistically representing a three-dimensional (3D) workspace that provides information feedback to the remote operator. Workspace representation methods that are currently in use have certain limitations - they are time consuming and labor intensive and require high-performance computers. A major objective of this study is the development of an efficient means of representing a workspace in 3D that has the capacity to provide interactive visual feedback to the operator of remote-controlled construction machines. To achieve this objective, the ability is required to acquire dense, accurate, and visually realistic 3D data that can be converted into high-quality models. This allows creation of a realistic 3D workspace representation of terrain, including objects that might be in proximity to the machine. For this purpose, this study proposes a multiimaging sensor data fusion-based system employing joint bilateral upsampling, which enhances the quality and availability of the information acquired in such an environment. The field experiment results show that combining data acquired with a complementary multiimaging sensor setup allows enhancement of the quality of the information available to the remote operator. The resulting task-specific 3D workspace representation can be successfully incorporated into the development of remote-controlled construction machines that require interactive visual feedback. This provides the opportunity to make human - machine interaction more efficient and to improve the remote operation capability by assisting the operator. Moreover, the proposed multiimaging sensor data fusion-based 3D representation approach has the potential for effective use in a broad class of applications within the construction industry.