  The contractive auto-encoder learns a representation of the input data that
captures the local manifold structure around each data point, through the
leading singular vectors of the Jacobian of the transformation from input to
representation. The corresponding singular values specify how much local
variation is plausible in directions associated with the corresponding singular
vectors, while remaining in a high-density region of the input space. This
paper proposes a procedure for generating samples that are consistent with the
local structure captured by a contractive auto-encoder. The associated
stochastic process defines a distribution from which one can sample, and which
experimentally appears to converge quickly and mix well between modes, compared
to Restricted Boltzmann Machines and Deep Belief Networks. The intuitions
behind this procedure can also be used to train the second layer of contraction
that pools lower-level features and learns to be invariant to the local
directions of variation discovered in the first layer. We show that this can
help learn and represent invariances present in the data and improve
classification error.
