  Although deep generative models have gained a lot of attention, most of the
existing works are designed for unimodal generation. In this paper, we explore
a new method for unconditional image-text pair generation. We design Multimodal
Cross-Quantization VAE (MXQ-VAE), a novel vector quantizer for joint image-text
representations, with which we discover that a joint image-text representation
space is effective for semantically consistent image-text pair generation. To
learn a multimodal semantic correlation in a quantized space, we combine VQ-VAE
with a Transformer encoder and apply an input masking strategy. Specifically,
MXQ-VAE accepts a masked image-text pair as input and learns a quantized joint
representation space, so that the input can be converted to a unified code
sequence, then we perform unconditional image-text pair generation with the
code sequence. Extensive experiments show the correlation between the quantized
joint space and the multimodal generation capability on synthetic and
real-world datasets. In addition, we demonstrate the superiority of our
approach in these two aspects over several baselines. The source code is
publicly available at: https://github.com/ttumyche/MXQ-VAE.
