  In this work, we present an end-to-end binaural speech synthesis system that
combines a low-bitrate audio codec with a powerful binaural decoder that is
capable of accurate speech binauralization while faithfully reconstructing
environmental factors like ambient noise or reverb. The network is a modified
vector-quantized variational autoencoder, trained with several carefully
designed objectives, including an adversarial loss. We evaluate the proposed
system on an internal binaural dataset with objective metrics and a perceptual
study. Results show that the proposed approach matches the ground truth data
more closely than previous methods. In particular, we demonstrate the
capability of the adversarial loss in capturing environment effects needed to
create an authentic auditory scene.
