  Deep neural networks with discrete latent variables offer the promise of
better symbolic reasoning, and learning abstractions that are more useful to
new tasks. There has been a surge in interest in discrete latent variable
models, however, despite several recent improvements, the training of discrete
latent variable models has remained challenging and their performance has
mostly failed to match their continuous counterparts. Recent work on vector
quantized autoencoders (VQ-VAE) has made substantial progress in this
direction, with its perplexity almost matching that of a VAE on datasets such
as CIFAR-10. In this work, we investigate an alternate training technique for
VQ-VAE, inspired by its connection to the Expectation Maximization (EM)
algorithm. Training the discrete bottleneck with EM helps us achieve better
image generation results on CIFAR-10, and together with knowledge distillation,
allows us to develop a non-autoregressive machine translation model whose
accuracy almost matches a strong greedy autoregressive baseline Transformer,
while being 3.3 times faster at inference.
