  The ability of deep neural networks to generalize well in the
overparameterized regime has become a subject of significant research interest.
We show that overparameterized autoencoders exhibit memorization, a form of
inductive bias that constrains the functions learned through the optimization
process to concentrate around the training examples, although the network could
in principle represent a much larger function class. In particular, we prove
that single-layer fully-connected autoencoders project data onto the
(nonlinear) span of the training examples. In addition, we show that deep
fully-connected autoencoders learn a map that is locally contractive at the
training examples, and hence iterating the autoencoder results in convergence
to the training examples. Finally, we prove that depth is necessary and provide
empirical evidence that it is also sufficient for memorization in convolutional
autoencoders. Understanding this inductive bias may shed light on the
generalization properties of overparametrized deep neural networks that are
currently unexplained by classical statistical theory.
