  We present a framework for modeling interactional communication in dyadic
conversations: given multimodal inputs of a speaker, we autoregressively output
multiple possibilities of corresponding listener motion. We combine the motion
and speech audio of the speaker using a motion-audio cross attention
transformer. Furthermore, we enable non-deterministic prediction by learning a
discrete latent representation of realistic listener motion with a novel
motion-encoding VQ-VAE. Our method organically captures the multimodal and
non-deterministic nature of nonverbal dyadic interactions. Moreover, it
produces realistic 3D listener facial motion synchronous with the speaker (see
video). We demonstrate that our method outperforms baselines qualitatively and
quantitatively via a rich suite of experiments. To facilitate this line of
research, we introduce a novel and large in-the-wild dataset of dyadic
conversations. Code, data, and videos available at
https://evonneng.github.io/learning2listen/.
