  We study the loss landscape of both shallow and deep, mildly
overparameterized ReLU neural networks on a generic finite input dataset for
the squared error loss. We show both by count and volume that most activation
patterns correspond to parameter regions with no bad local minima. Furthermore,
for one-dimensional input data, we show most activation regions realizable by
the network contain a high dimensional set of global minima and no bad local
minima. We experimentally confirm these results by finding a phase transition
from most regions having full rank Jacobian to many regions having deficient
rank depending on the amount of overparameterization.
