  It is widely conjectured that the reason that training algorithms for neural
networks are successful because all local minima lead to similar performance,
for example, see (LeCun et al., 2015, Choromanska et al., 2015, Dauphin et al.,
2014). Performance is typically measured in terms of two metrics: training
performance and generalization performance. Here we focus on the training
performance of single-layered neural networks for binary classification, and
provide conditions under which the training error is zero at all local minima
of a smooth hinge loss function. Our conditions are roughly in the following
form: the neurons have to be strictly convex and the surrogate loss function
should be a smooth version of hinge loss. We also provide counterexamples to
show that when the loss function is replaced with quadratic loss or logistic
loss, the result may not hold.
