  Face frontalization consists of synthesizing a frontally-viewed face from an
arbitrarily-viewed one. The main contribution of this paper is a frontalization
methodology that preserves non-rigid facial deformations in order to boost the
performance of visually assisted speech communication. The method alternates
between the estimation of (i)~the rigid transformation (scale, rotation, and
translation) and (ii)~the non-rigid deformation between an arbitrarily-viewed
face and a face model. The method has two important merits: it can deal with
non-Gaussian errors in the data and it incorporates a dynamical face
deformation model. For that purpose, we use the generalized Student
t-distribution in combination with a linear dynamic system in order to account
for both rigid head motions and time-varying facial deformations caused by
speech production. We propose to use the zero-mean normalized cross-correlation
(ZNCC) score to evaluate the ability of the method to preserve facial
expressions. The method is thoroughly evaluated and compared with several state
of the art methods, either based on traditional geometric models or on deep
learning. Moreover, we show that the method, when incorporated into deep
learning pipelines, namely lip reading and speech enhancement, improves word
recognition and speech intelligibilty scores by a considerable margin.
Supplemental material is accessible at
https://team.inria.fr/robotlearn/research/facefrontalization/
