  In this paper, we suggest a framework to make use of mutual information as a
regularization criterion to train Auto-Encoders (AEs). In the proposed
framework, AEs are regularized by minimization of the mutual information
between input and encoding variables of AEs during the training phase. In order
to estimate the entropy of the encoding variables and the mutual information,
we propose a non-parametric method. We also give an information theoretic view
of Variational AEs (VAEs), which suggests that VAEs can be considered as
parametric methods that estimate entropy. Experimental results show that the
proposed non-parametric models have more degree of freedom in terms of
representation learning of features drawn from complex distributions such as
Mixture of Gaussians, compared to methods which estimate entropy using
parametric approaches, such as Variational AEs.
