  This work seeks the possibility of generating the human face from voice
solely based on the audio-visual data without any human-labeled annotations. To
this end, we propose a multi-modal learning framework that links the inference
stage and generation stage. First, the inference networks are trained to match
the speaker identity between the two different modalities. Then the trained
inference networks cooperate with the generation network by giving conditional
information about the voice. The proposed method exploits the recent
development of GANs techniques and generates the human face directly from the
speech waveform making our system fully end-to-end. We analyze the extent to
which the network can naturally disentangle two latent factors that contribute
to the generation of a face image - one that comes directly from a speech
signal and the other that is not related to it - and explore whether the
network can learn to generate natural human face image distribution by modeling
these factors. Experimental results show that the proposed network can not only
match the relationship between the human face and speech, but can also generate
the high-quality human face sample conditioned on its speech. Finally, the
correlation between the generated face and the corresponding speech is
quantitatively measured to analyze the relationship between the two modalities.
