  We propose a novel method to use both audio and a low-resolution image to
perform extreme face super-resolution (a 16x increase of the input size). When
the resolution of the input image is very low (e.g., 8x8 pixels), the loss of
information is so dire that important details of the original identity have
been lost and audio can aid the recovery of a plausible high-resolution image.
In fact, audio carries information about facial attributes, such as gender and
age. To combine the aural and visual modalities, we propose a method to first
build the latent representations of a face from the lone audio track and then
from the lone low-resolution image. We then train a network to fuse these two
representations. We show experimentally that audio can assist in recovering
attributes such as the gender, the age and the identity, and thus improve the
correctness of the high-resolution image reconstruction process. Our procedure
does not make use of human annotation and thus can be easily trained with
existing video datasets. Moreover, we show that our model builds a factorized
representation of images and audio as it allows one to mix low-resolution
images and audio from different videos and to generate realistic faces with
semantically meaningful combinations.
