  Information theoretic quantities play an important role in various settings
in machine learning, including causality testing, structure inference in
graphical models, time-series problems, feature selection as well as in
providing privacy guarantees. A key quantity of interest is the mutual
information and generalizations thereof, including conditional mutual
information, multivariate mutual information, total correlation and directed
information. While the aforementioned information quantities are well defined
in arbitrary probability spaces, existing estimators add or subtract entropies
(we term them $\Sigma H$ methods). These methods work only in purely discrete
space or purely continuous case since entropy (or differential entropy) is well
defined only in that regime.
  In this paper, we define a general graph divergence measure ($\mathbb{GDM}$),
as a measure of incompatibility between the observed distribution and a given
graphical model structure. This generalizes the aforementioned information
measures and we construct a novel estimator via a coupling trick that directly
estimates these multivariate information measures using the Radon-Nikodym
derivative. These estimators are proven to be consistent in a general setting
which includes several cases where the existing estimators fail, thus providing
the only known estimators for the following settings: (1) the data has some
discrete and some continuous-valued components (2) some (or all) of the
components themselves are discrete-continuous mixtures (3) the data is
real-valued but does not have a joint density on the entire space, rather is
supported on a low-dimensional manifold. We show that our proposed estimators
significantly outperform known estimators on synthetic and real datasets.
