  Frustrating text entry interface has been a major obstacle in participating
in social activities in augmented reality (AR). Popular options, such as
mid-air keyboard interface, wireless keyboards or voice input, either suffer
from poor ergonomic design, limited accuracy, or are simply embarrassing to use
in public. This paper proposes and validates a deep-learning based approach,
that enables AR applications to accurately predict keystrokes from the user
perspective RGB video stream that can be captured by any AR headset. This
enables a user to perform typing activities on any flat surface and eliminates
the need of a physical or virtual keyboard. A two-stage model, combing an
off-the-shelf hand landmark extractor and a novel adaptive Convolutional
Recurrent Neural Network (C-RNN), was trained using our newly built dataset.
The final model was capable of adaptive processing user-perspective video
streams at ~32 FPS. This base model achieved an overall accuracy of $91.05\%$
when typing 40 Words per Minute (wpm), which is how fast an average person
types with two hands on a physical keyboard. The Normalised Levenshtein
Distance also further confirmed the real-world applicability of that our
approach. The promising results highlight the viability of our approach and the
potential for our method to be integrated into various applications. We also
discussed the limitations and future research required to bring such technique
into a production system.
