  Generation of photo-realistic images, semantic editing and representation
learning are a few of many potential applications of high resolution generative
models. Recent progress in GANs have established them as an excellent choice
for such tasks. However, since they do not provide an inference model, image
editing or downstream tasks such as classification can not be done on real
images using the GAN latent space. Despite numerous efforts to train an
inference model or design an iterative method to invert a pre-trained
generator, previous methods are dataset (e.g. human face images) and
architecture (e.g. StyleGAN) specific. These methods are nontrivial to extend
to novel datasets or architectures. We propose a general framework that is
agnostic to architecture and datasets. Our key insight is that, by training the
inference and the generative model together, we allow them to adapt to each
other and to converge to a better quality model. Our \textbf{InvGAN}, short for
Invertible GAN, successfully embeds real images to the latent space of a high
quality generative model. This allows us to perform image inpainting, merging,
interpolation and online data augmentation. We demonstrate this with extensive
qualitative and quantitative experiments.
