  We developed an American Sign Language (ASL) learning platform in a Virtual
Reality (VR) environment to facilitate immersive interaction and real-time
feedback for ASL learners. We describe the first game to use an interactive
teaching style in which users learn from a fluent signing avatar and the first
implementation of ASL sign recognition using deep learning within the VR
environment. Advanced motion-capture technology powers an expressive ASL
teaching avatar within an immersive three-dimensional environment. The teacher
demonstrates an ASL sign for an object, prompting the user to copy the sign.
Upon the user's signing, a third-party plugin executes the sign recognition
process alongside a deep learning model. Depending on the accuracy of a user's
sign production, the avatar repeats the sign or introduces a new one. We
gathered a 3D VR ASL dataset from fifteen diverse participants to power the
sign recognition model. The proposed deep learning model's training,
validation, and test accuracy are 90.12%, 89.37%, and 86.66%, respectively. The
functional prototype can teach sign language vocabulary and be successfully
adapted as an interactive ASL learning platform in VR.
