  We present a real-time deep learning framework for video-based facial
performance capture -- the dense 3D tracking of an actor's face given a
monocular video. Our pipeline begins with accurately capturing a subject using
a high-end production facial capture pipeline based on multi-view stereo
tracking and artist-enhanced animations. With 5-10 minutes of captured footage,
we train a convolutional neural network to produce high-quality output,
including self-occluded regions, from a monocular video sequence of that
subject. Since this 3D facial performance capture is fully automated, our
system can drastically reduce the amount of labor involved in the development
of modern narrative-driven video games or films involving realistic digital
doubles of actors and potentially hours of animated dialogue per character. We
compare our results with several state-of-the-art monocular real-time facial
capture techniques and demonstrate compelling animation inference in
challenging areas such as eyes and lips.
