  We consider the optimization problem associated with training simple ReLU
neural networks of the form $\mathbf{x}\mapsto
\sum_{i=1}^{k}\max\{0,\mathbf{w}_i^\top \mathbf{x}\}$ with respect to the
squared loss. We provide a computer-assisted proof that even if the input
distribution is standard Gaussian, even if the dimension is arbitrarily large,
and even if the target values are generated by such a network, with orthonormal
parameter vectors, the problem can still have spurious local minima once $6\le
k\le 20$. By a concentration of measure argument, this implies that in high
input dimensions, \emph{nearly all} target networks of the relevant sizes lead
to spurious local minima. Moreover, we conduct experiments which show that the
probability of hitting such local minima is quite high, and increasing with the
network size. On the positive side, mild over-parameterization appears to
drastically reduce such local minima, indicating that an over-parameterization
assumption is necessary to get a positive result in this setting.
