  VR Facial Animation is necessary in applications requiring clear view of the
face, even though a VR headset is worn. In our case, we aim to animate the face
of an operator who is controlling our robotic avatar system. We propose a
real-time capable pipeline with very fast adaptation for specific operators. In
a quick enrollment step, we capture a sequence of source images from the
operator without the VR headset which contain all the important
operator-specific appearance information. During inference, we then use the
operator keypoint information extracted from a mouth camera and two eye cameras
to estimate the target expression and head pose, to which we map the appearance
of a source still image. In order to enhance the mouth expression accuracy, we
dynamically select an auxiliary expression frame from the captured sequence.
This selection is done by learning to transform the current mouth keypoints
into the source camera space, where the alignment can be determined accurately.
We, furthermore, demonstrate an eye tracking pipeline that can be trained in
less than a minute, a time efficient way to train the whole pipeline given a
dataset that includes only complete faces, show exemplary results generated by
our method, and discuss performance at the ANA Avatar XPRIZE semifinals.
