  Hand gestures are an intuitive, socially acceptable, and a non-intrusive
interaction modality in Mixed Reality (MR) and smartphone based applications.
Unlike speech interfaces, they tend to perform well even in shared and public
spaces. Hand gestures can also be used to interact with smartphones in
situations where the user's ability to physically touch the device is impaired.
However, accurate gesture recognition can be achieved through state-of-the-art
deep learning models or with the use of expensive sensors. Despite the
robustness of these deep learning models, they are computationally heavy and
memory hungry, and obtaining real-time performance on-device without additional
hardware is still a challenge. To address this, we propose AirPen: an analogue
to pen on paper, but in air, for in-air writing and gestural commands that
works seamlessly in First and Second Person View. The models are trained on a
GPU machine and ported on an Android smartphone. AirPen comprises of three deep
learning models that work in tandem: MobileNetV2 for hand localisation, our
custom fingertip regression architecture followed by a Bi-LSTM model for
gesture classification. The overall framework works in real-time on mobile
devices and achieves a classification accuracy of 80% with an average latency
of only 0.12 s.
