  Real-time recognition of dynamic hand gestures from video streams is a
challenging task since (i) there is no indication when a gesture starts and
ends in the video, (ii) performed gestures should only be recognized once, and
(iii) the entire architecture should be designed considering the memory and
power budget. In this work, we address these challenges by proposing a
hierarchical structure enabling offline-working convolutional neural network
(CNN) architectures to operate online efficiently by using sliding window
approach. The proposed architecture consists of two models: (1) A detector
which is a lightweight CNN architecture to detect gestures and (2) a classifier
which is a deep CNN to classify the detected gestures. In order to evaluate the
single-time activations of the detected gestures, we propose to use Levenshtein
distance as an evaluation metric since it can measure misclassifications,
multiple detections, and missing detections at the same time. We evaluate our
architecture on two publicly available datasets - EgoGesture and NVIDIA Dynamic
Hand Gesture Datasets - which require temporal detection and classification of
the performed hand gestures. ResNeXt-101 model, which is used as a classifier,
achieves the state-of-the-art offline classification accuracy of 94.04% and
83.82% for depth modality on EgoGesture and NVIDIA benchmarks, respectively. In
real-time detection and classification, we obtain considerable early detections
while achieving performances close to offline operation. The codes and
pretrained models used in this work are publicly available.
