Context-aware worker intent interpretation for human-robot collaboration in construction

With years of technical development, construction robots and/or autonomous machines have shown the potential to enhance productivity and safety in the construction industry. However, robots and autonomous machines have not been widely adopted on construction sites. There are various reasons contributing to the low adoption of robots in construction including operational and personnel barriers. An intuitive and accurate human intent representation can help contribute to resolving the above barriers. On one hand, the establishment of such representation can greatly enhance the operability of robots in dynamic construction environments. On the other hand, such representation helps to build a safe environment for worker-robot collaboration. So far, there are much related work about human intent interpretation, such as recognizing human actions, identifying hand gestures, automating eye tracking, and understanding speech language. These references show that the recent advance in technologies (e.g., computer vision, wearable sensor) has built a solid foundation to interpret and predict the intentions of onsite construction workers and support their collaborations with robotic machines. However, two research questions need to be answered before making such interpretations and predictions work well on construction sites: (1) How to capture and interpret worker intents accurately for worker-robot collaborations in construction? (2) How to extract useful context information to facilitate representation on construction sites? To answer these two questions, a context-aware human intent representation is proposed in this study to support human-robot collaboration on construction sites. It consists of three components: recognition building, object-enhanced interaction and machine-aware collaboration. In the first component recognition building, a novel vision-based method is developed to achieve gesture recognition. Since computer vision technologies may be easily affected by the construction environment (e.g., diverse dust and light conditions), a novel wearable sensors-based method is then developed. Through a comparison between these two methods, the sensor-based method is found to have the advantages of early triggering and robust anti-interference capabilities, but may incur higher communication costs in human-robot interactions.In the second component object-enhanced interaction, a novel object-aware method is proposed for human-robot collaboration in construction, integrating first-person vision and gesture recognition. An end-to-end two-stream network which includes a first-person view-based stream and a motion sensory data-based stream is designed. The first-person view-based stream models the user’s gaze using an attention module to concentrate on the important spatiotemporal regions of first-person video for context extraction. The motion sensory data-based stream is used to process the motion sensory data to extract features related to the hand motions. Finally, the feature maps coming from these two streams are fused to achieve the hand gesture recognition.In the third component machine-aware collaboration, a novel machine-aware hand gesture recognition method is developed as a human-robot interface for use on construction sites having multiple types of machines. The developed method firstly relies on an eye tracker to visually detect and track construction machines in the first-person view. Then, the machine-of-interest is determined based on the bounding boxes of machines and gaze points. Finally, a hand gesture recognition architecture is incorporated with the machine information for conveying messages to the machine-of-interest. The above methods have been evaluated and tested by experiments on different construction sites. The evaluation results have demonstrated that the proposed methods can capture and interpret the worker intents accurately with context awareness to support human-robot collaborations on construction sites. The expected contributions of the proposed methods include improving interaction eff ciency with construction robots, decreasing onsite safety issues, refining the design and implementation of construction robots, promoting the adoption of robots in construction, etc. Future work will focus on the following aspects. First, expanding the dataset to encompass a wider range of construction sites, subjects, equipment, tasks, and types of hand gestures will enhance the robustness of the proposed methods. Second, efforts will be made to address the technological challenges associated with automation, including electronics, computation, and communication. Third, the integration of sensor and data fusion techniques will be explored to enhance the reliability of message communication between workers and machines. Fourth, gathering feedback from workers on novel human-robot interfaces will be prioritized to evaluate their trust in construction robots and potential safety concerns during human-robot interactions on construction sites.