  Mutual Information (MI) has been widely used as a loss regularizer for
training neural networks. This has been particularly effective when learn
disentangled or compressed representations of high dimensional data. However,
differential entropy (DE), another fundamental measure of information, has not
found widespread use in neural network training. Although DE offers a
potentially wider range of applications than MI, off-the-shelf DE estimators
are either non differentiable, computationally intractable or fail to adapt to
changes in the underlying distribution. These drawbacks prevent them from being
used as regularizers in neural networks training. To address shortcomings in
previously proposed estimators for DE, here we introduce KNIFE, a fully
parameterized, differentiable kernel-based estimator of DE. The flexibility of
our approach also allows us to construct KNIFE-based estimators for conditional
(on either discrete or continuous variables) DE, as well as MI. We empirically
validate our method on high-dimensional synthetic data and further apply it to
guide the training of neural networks for real-world tasks. Our experiments on
a large variety of tasks, including visual domain adaptation, textual fair
classification, and textual fine-tuning demonstrate the effectiveness of
KNIFE-based estimation. Code can be found at
https://github.com/g-pichler/knife.
