  Deep generative models have made tremendous advances in image and signal
representation learning and generation. These models employ the full Euclidean
space or a bounded subset as the latent space, whose flat geometry, however, is
often too simplistic to meaningfully reflect the manifold structure of the
data. In this work, we advocate the use of a multi-chart latent space for
better data representation. Inspired by differential geometry, we propose a
\textbf{Chart Auto-Encoder (CAE)} and prove a universal approximation theorem
on its representation capability. We show that the training data size and the
network size scale exponentially in approximation error with an exponent
depending on the intrinsic dimension of the data manifold. CAE admits desirable
manifold properties that auto-encoders with a flat latent space fail to obey,
predominantly proximity of data. We conduct extensive experimentation with
synthetic and real-life examples to demonstrate that CAE provides
reconstruction with high fidelity, preserves proximity in the latent space, and
generates new data remaining near the manifold. These experiments show that CAE
is advantageous over existing auto-encoders and variants by preserving the
topology of the data manifold as well as its geometry.
