  Hand gestures form an intuitive means of interaction in Mixed Reality (MR)
applications. However, accurate gesture recognition can be achieved only
through state-of-the-art deep learning models or with the use of expensive
sensors. Despite the robustness of these deep learning models, they are
generally computationally expensive and obtaining real-time performance
on-device is still a challenge. To this end, we propose a novel lightweight
hand gesture recognition framework that works in First Person View for wearable
devices. The models are trained on a GPU machine and ported on an Android
smartphone for its use with frugal wearable devices such as the Google
Cardboard and VR Box. The proposed hand gesture recognition framework is driven
by a cascade of state-of-the-art deep learning models: MobileNetV2 for hand
localisation, our custom fingertip regression architecture followed by a
Bi-LSTM model for gesture classification. We extensively evaluate the framework
on our EgoGestAR dataset. The overall framework works in real-time on mobile
devices and achieves a classification accuracy of 80% on EgoGestAR video
dataset with an average latency of only 0.12 s.
