  Why do deep neural networks (DNNs) benefit from very high dimensional
parameter spaces? Their huge parameter complexities vs. stunning performances
in practice is all the more intriguing and not explainable using the standard
theory of model selection for regular models. In this work, we propose a
geometrically flavored information-theoretic approach to study this phenomenon.
Namely, we introduce the locally varying dimensionality of the parameter space
of neural network models by considering the number of significant dimensions of
the Fisher information matrix, and model the parameter space as a manifold
using the framework of singular semi-Riemannian geometry. We derive model
complexity measures which yield short description lengths for deep neural
network models based on their singularity analysis thus explaining the good
performance of DNNs despite their large number of parameters.
