  For bidirectional joint image-text modeling, we develop variational
hetero-encoder (VHE) randomized generative adversarial network (GAN), a
versatile deep generative model that integrates a probabilistic text decoder,
probabilistic image encoder, and GAN into a coherent end-to-end multi-modality
learning framework. VHE randomized GAN (VHE-GAN) encodes an image to decode its
associated text, and feeds the variational posterior as the source of
randomness into the GAN image generator. We plug three off-the-shelf modules,
including a deep topic model, a ladder-structured image encoder, and
StackGAN++, into VHE-GAN, which already achieves competitive performance. This
further motivates the development of VHE-raster-scan-GAN that generates
photo-realistic images in not only a multi-scale low-to-high-resolution manner,
but also a hierarchical-semantic coarse-to-fine fashion. By capturing and
relating hierarchical semantic and visual concepts with end-to-end training,
VHE-raster-scan-GAN achieves state-of-the-art performance in a wide variety of
image-text multi-modality learning and generation tasks.
