  We define a notion of complexity, which quantifies the nonlinearity of the
computation of a neural network, as well as a complementary measure of the
effective dimension of feature representations. We investigate these
observables both for trained networks for various datasets as well as explore
their dynamics during training, uncovering in particular power law scaling.
These observables can be understood in a dual way as uncovering hidden internal
structure of the datasets themselves as a function of scale or depth. The
entropic character of the proposed notion of complexity should allow to
transfer modes of analysis from neuroscience and statistical physics to the
domain of artificial neural networks. The introduced observables can be applied
without any change to the analysis of biological neuronal systems.
