  When it comes to addressing the safety/security related needs at different
production/construction sites, accurate detection of the presence of workers,
vehicles, equipment important and formed an integral part of computer
vision-based surveillance systems (CVSS). Traditional CVSS systems focus on the
use of different computer vision and pattern recognition algorithms overly
reliant on manual extraction of features and small datasets, limiting their
usage because of low accuracy, need for expert knowledge and high computational
costs. The main objective of this paper is to provide decision makers at sites
with a practical yet comprehensive deep learning and IoT based solution to
tackle various computer vision related problems such as scene classification,
object detection in scenes, semantic segmentation, scene captioning etc. Our
overarching goal is to address the central question of What is happening at
this site and where is it happening in an automated fashion minimizing the need
for human resources dedicated to surveillance. We developed Deep ExxonMobil Eye
for Video Analysis (DEEVA) package to handle scene classification, object
detection, semantic segmentation and captioning of scenes in a hierarchical
approach. The results reveal that transfer learning with the RetinaNet object
detector is able to detect the presence of workers, different types of
vehicles/construction equipment, safety related objects at a high level of
accuracy (above 90%). With the help of deep learning to automatically extract
features and IoT technology to automatic capture, transfer and process vast
amount of realtime images, this framework is an important step towards the
development of intelligent surveillance systems aimed at addressing myriads of
open ended problems in the realm of security/safety monitoring, productivity
assessments and future decision making.
