  Most entropy measures depend on the spread of the probability distribution
over the sample space $\mathcal{X}$, and the maximum entropy achievable scales
proportionately with the sample space cardinality $|\mathcal{X}|$. For a finite
$|\mathcal{X}|$, this yields robust entropy measures which satisfy many
important properties, such as invariance to bijections, while the same is not
true for continuous spaces (where $|\mathcal{X}|=\infty$). Furthermore, since
$\mathbb{R}$ and $\mathbb{R}^d$ ($d\in \mathbb{Z}^+$) have the same cardinality
(from Cantor's correspondence argument), cardinality-dependent entropy measures
cannot encode the data dimensionality. In this work, we question the role of
cardinality and distribution spread in defining entropy measures for continuous
spaces, which can undergo multiple rounds of transformations and distortions,
e.g., in neural networks. We find that the average value of the local intrinsic
dimension of a distribution, denoted as ID-Entropy, can serve as a robust
entropy measure for continuous spaces, while capturing the data dimensionality.
We find that ID-Entropy satisfies many desirable properties and can be extended
to conditional entropy, joint entropy and mutual-information variants.
ID-Entropy also yields new information bottleneck principles and also links to
causality. In the context of deep learning, for feedforward architectures, we
show, theoretically and empirically, that the ID-Entropy of a hidden layer
directly controls the generalization gap for both classifiers and
auto-encoders, when the target function is Lipschitz continuous. Our work
primarily shows that, for continuous spaces, taking a structural rather than a
statistical approach yields entropy measures which preserve intrinsic data
dimensionality, while being relevant for studying various architectures.
