  With the recent advancements in deep learning and computer vision, the
AI-powered construction machine such as autonomous excavator has made
significant progress. Safety is the most important section in modern
construction, where construction machines are more and more automated. In this
paper, we propose a vision-based excavator perception, activity analysis, and
safety monitoring system. Our perception system could detect multi-class
construction machines and humans in real-time while estimating the poses and
actions of the excavator. Then, we present a novel safety monitoring and
excavator activity analysis system based on the perception result. To evaluate
the performance of our method, we collect a dataset using the Autonomous
Excavator System (AES) including multi-class of objects in different lighting
conditions with human annotations. We also evaluate our method on a benchmark
construction dataset. The results showed our YOLO v5 multi-class objects
detection model improved inference speed by 8 times (YOLO v5 x-large) to 34
times (YOLO v5 small) compared with Faster R-CNN/ YOLO v3 model. Furthermore,
the accuracy of YOLO v5 models is improved by 2.7% (YOLO v5 x-large) while
model size is reduced by 63.9% (YOLO v5 x-large) to 93.9% (YOLO v5 small). The
experimental results show that the proposed action recognition approach
outperforms the state-of-the-art approaches on top-1 accuracy by about 5.18%.
The proposed real-time safety monitoring system is not only designed for our
Autonomous Excavator System (AES) in solid waste scenes, it can also be applied
to general construction scenarios.
