  Regularization is a big issue for training deep neural networks. In this
paper, we propose a new information-theory-based regularization scheme named
SHADE for SHAnnon DEcay. The originality of the approach is to define a prior
based on conditional entropy, which explicitly decouples the learning of
invariant representations in the regularizer and the learning of correlations
between inputs and labels in the data fitting term. Our second contribution is
to derive a stochastic version of the regularizer compatible with deep
learning, resulting in a tractable training scheme. We empirically validate the
efficiency of our approach to improve classification performances compared to
common regularization schemes on several standard architectures.
