  We present an encoder-powered generative adversarial network (EncGAN) that is
able to learn both the multi-manifold structure and the abstract features of
data. Unlike the conventional decoder-based GANs, EncGAN uses an encoder to
model the manifold structure and invert the encoder to generate data. This
unique scheme enables the proposed model to exclude discrete features from the
smooth structure modeling and learn multi-manifold data without being hindered
by the disconnections. Also, as EncGAN requires a single latent space to carry
the information for all the manifolds, it builds abstract features shared among
the manifolds in the latent space. For an efficient computation, we formulate
EncGAN using a simple regularizer, and mathematically prove its validity. We
also experimentally demonstrate that EncGAN successfully learns the
multi-manifold structure and the abstract features of MNIST, 3D-chair and
UT-Zap50k datasets. Our analysis shows that the learned abstract features are
disentangled and make a good style-transfer even when the source data is off
the trained distribution.
