  Facial alignment involves finding a set of landmark points on an image with a
known semantic meaning. However, this semantic meaning of landmark points is
often lost in 2D approaches where landmarks are either moved to visible
boundaries or ignored as the pose of the face changes. In order to extract
consistent alignment points across large poses, the 3D structure of the face
must be considered in the alignment step. However, extracting a 3D structure
from a single 2D image usually requires alignment in the first place. We
present our novel approach to simultaneously extract the 3D shape of the face
and the semantically consistent 2D alignment through a 3D Spatial Transformer
Network (3DSTN) to model both the camera projection matrix and the warping
parameters of a 3D model. By utilizing a generic 3D model and a Thin Plate
Spline (TPS) warping function, we are able to generate subject specific 3D
shapes without the need for a large 3D shape basis. In addition, our proposed
network can be trained in an end-to-end framework on entirely synthetic data
from the 300W-LP dataset. Unlike other 3D methods, our approach only requires
one pass through the network resulting in a faster than real-time alignment.
Evaluations of our model on the Annotated Facial Landmarks in the Wild (AFLW)
and AFLW2000-3D datasets show our method achieves state-of-the-art performance
over other 3D approaches to alignment.
