  We explore some mathematical features of the loss landscape of
overparameterized neural networks. A priori one might imagine that the loss
function looks like a typical function from $\mathbb{R}^n$ to $\mathbb{R}$ - in
particular, nonconvex, with discrete global minima. In this paper, we prove
that in at least one important way, the loss function of an overparameterized
neural network does not look like a typical function. If a neural net has $n$
parameters and is trained on $d$ data points, with $n>d$, we show that the
locus $M$ of global minima of $L$ is usually not discrete, but rather an $n-d$
dimensional submanifold of $\mathbb{R}^n$. In practice, neural nets commonly
have orders of magnitude more parameters than data points, so this observation
implies that $M$ is typically a very high-dimensional subset of $\mathbb{R}^n$.
