  Autoencoders are a type of unsupervised neural networks, which can be used to
solve various tasks, e.g., dimensionality reduction, image compression, and
image denoising. An AE has two goals: (i) compress the original input to a
low-dimensional space at the bottleneck of the network topology using an
encoder, (ii) reconstruct the input from the representation at the bottleneck
using a decoder. Both encoder and decoder are optimized jointly by minimizing a
distortion-based loss which implicitly forces the model to keep only those
variations of input data that are required to reconstruct the and to reduce
redundancies. In this paper, we propose a scheme to explicitly penalize feature
redundancies in the bottleneck representation. To this end, we propose an
additional loss term, based on the pair-wise correlation of the neurons, which
complements the standard reconstruction loss forcing the encoder to learn a
more diverse and richer representation of the input. We tested our approach
across different tasks: dimensionality reduction using three different dataset,
image compression using the MNIST dataset, and image denoising using fashion
MNIST. The experimental results show that the proposed loss leads consistently
to superior performance compared to the standard AE loss.
