  Teleconference or telepresence based on virtual reality (VR) headmount
display (HMD) device is a very interesting and promising application since HMD
can provide immersive feelings for users. However, in order to facilitate
face-to-face communications for HMD users, real-time 3D facial performance
capture of a person wearing HMD is needed, which is a very challenging task due
to the large occlusion caused by HMD. The existing limited solutions are very
complex either in setting or in approach as well as lacking the performance
capture of 3D eye gaze movement. In this paper, we propose a convolutional
neural network (CNN) based solution for real-time 3D face-eye performance
capture of HMD users without complex modification to devices. To address the
issue of lacking training data, we generate massive pairs of HMD face-label
dataset by data synthesis as well as collecting VR-IR eye dataset from multiple
subjects. Then, we train a dense-fitting network for facial region and an eye
gaze network to regress 3D eye model parameters. Extensive experimental results
demonstrate that our system can efficiently and effectively produce in real
time a vivid personalized 3D avatar with the correct identity, pose, expression
and eye motion corresponding to the HMD user.
