  Mutual Information (MI) plays an important role in representation learning.
However, MI is unfortunately intractable in continuous and high-dimensional
settings. Recent advances establish tractable and scalable MI estimators to
discover useful representation. However, most of the existing methods are not
capable of providing an accurate estimation of MI with low-variance when the MI
is large. We argue that directly estimating the gradients of MI is more
appealing for representation learning than estimating MI in itself. To this
end, we propose the Mutual Information Gradient Estimator (MIGE) for
representation learning based on the score estimation of implicit
distributions. MIGE exhibits a tight and smooth gradient estimation of MI in
the high-dimensional and large-MI settings. We expand the applications of MIGE
in both unsupervised learning of deep representations based on InfoMax and the
Information Bottleneck method. Experimental results have indicated significant
performance improvement in learning useful representation.
