  We introduce the "adversarial code learning" (ACL) module that improves
overall image generation performance to several types of deep models. Instead
of performing a posterior distribution modeling in the pixel spaces of
generators, ACLs aim to jointly learn a latent code with another image
encoder/inference net, with a prior noise as its input. We conduct the learning
in an adversarial learning process, which bears a close resemblance to the
original GAN but again shifts the learning from image spaces to prior and
latent code spaces. ACL is a portable module that brings up much more
flexibility and possibilities in generative model designs. First, it allows
flexibility to convert non-generative models like Autoencoders and standard
classification models to decent generative models. Second, it enhances existing
GANs' performance by generating meaningful codes and images from any part of
the prior. We have incorporated our ACL module with the aforementioned
frameworks and have performed experiments on synthetic, MNIST, CIFAR-10, and
CelebA datasets. Our models have achieved significant improvements which
demonstrated the generality for image generation tasks.
