  While GAN is a powerful model for generating images, its inability to infer a
latent space directly limits its use in applications requiring an encoder. Our
paper presents a simple architectural setup that combines the generative
capabilities of GAN with an encoder. We accomplish this by combining the
encoder with the discriminator using shared weights, then training them
simultaneously using a new loss term. We model the output of the encoder latent
space via a GMM, which leads to both good clustering using this latent space
and improved image generation by the GAN. Our framework is generic and can be
easily plugged into any GAN strategy. In particular, we demonstrate it both
with Vanilla GAN and Wasserstein GAN, where in both it leads to an improvement
in the generated images in terms of both the IS and FID scores. Moreover, we
show that our encoder learns a meaningful representation as its clustering
results are competitive with the current GAN-based state-of-the-art in
clustering.
