  Visualization is a crucial step in exploratory data analysis. One possible
approach is to train an autoencoder with low-dimensional latent space. Large
network depth and width can help unfolding the data. However, such expressive
networks can achieve low reconstruction error even when the latent
representation is distorted. To avoid such misleading visualizations, we
propose first a differential geometric perspective on the decoder, leading to
insightful diagnostics for an embedding's distortion, and second a new
regularizer mitigating such distortion. Our ``Geometric Autoencoder'' avoids
stretching the embedding spuriously, so that the visualization captures the
data structure more faithfully. It also flags areas where little distortion
could not be achieved, thus guarding against misinterpretation.
