  Mutual information is a general statistical dependency measure which has
found applications in representation learning, causality, domain generalization
and computational biology. However, mutual information estimators are typically
evaluated on simple families of probability distributions, namely multivariate
normal distribution and selected distributions with one-dimensional random
variables. In this paper, we show how to construct a diverse family of
distributions with known ground-truth mutual information and propose a
language-independent benchmarking platform for mutual information estimators.
We discuss the general applicability and limitations of classical and neural
estimators in settings involving high dimensions, sparse interactions,
long-tailed distributions, and high mutual information. Finally, we provide
guidelines for practitioners on how to select appropriate estimator adapted to
the difficulty of problem considered and issues one needs to consider when
applying an estimator to a new data set.
