  Suitable lateral connections between encoder and decoder are shown to allow
higher layers of a denoising autoencoder (dAE) to focus on invariant
representations. In regular autoencoders, detailed information needs to be
carried through the highest layers but lateral connections from encoder to
decoder relieve this pressure. It is shown that abstract invariant features can
be translated to detailed reconstructions when invariant features are allowed
to modulate the strength of the lateral connection. Three dAE structures with
modulated and additive lateral connections, and without lateral connections
were compared in experiments using real-world images. The experiments verify
that adding modulated lateral connections to the model 1) improves the accuracy
of the probability model for inputs, as measured by denoising performance; 2)
results in representations whose degree of invariance grows faster towards the
higher layers; and 3) supports the formation of diverse invariant poolings.
