  The learnability of different neural architectures can be characterized
directly by computable measures of data complexity. In this paper, we reframe
the problem of architecture selection as understanding how data determines the
most expressive and generalizable architectures suited to that data, beyond
inductive bias. After suggesting algebraic topology as a measure for data
complexity, we show that the power of a network to express the topological
complexity of a dataset in its decision region is a strictly limiting factor in
its ability to generalize. We then provide the first empirical characterization
of the topological capacity of neural networks. Our empirical analysis shows
that at every level of dataset complexity, neural networks exhibit topological
phase transitions. This observation allowed us to connect existing theory to
empirically driven conjectures on the choice of architectures for
fully-connected neural networks.
