  To improve the experiences of face-to-face conversation with avatar, this
paper presents a novel conversation system. It is composed of two
sequence-to-sequence models respectively for listening and speaking and a
Generative Adversarial Network (GAN) based realistic avatar synthesizer. The
models exploit the facial action and head pose to learn natural human
reactions. Based on the models' output, the synthesizer uses the Pixel2Pixel
model to generate realistic facial images. To show the improvement of our
system, we use a 3D model based avatar driving scheme as a reference. We train
and evaluate our neural networks with the data from ESPN shows. Experimental
results show that our conversation system can generate natural facial reactions
and realistic facial images.
