  Neural networks are usually trained with different variants of gradient
descent based optimization algorithms such as stochastic gradient descent or
the Adam optimizer. Recent theoretical work states that the critical points
(where the gradient of the loss is zero) of two-layer ReLU networks with the
square loss are not all local minima. However, in this work we will explore an
algorithm for training two-layer neural networks with ReLU-like activation and
the square loss that alternatively finds the critical points of the loss
function analytically for one layer while keeping the other layer and the
neuron activation pattern fixed. Experiments indicate that this simple
algorithm can find deeper optima than Stochastic Gradient Descent or the Adam
optimizer, obtaining significantly smaller training loss values on four out of
the five real datasets evaluated. Moreover, the method is faster than the
gradient descent methods and has virtually no tuning parameters.
