  The optimization problem behind neural networks is highly non-convex.
Training with stochastic gradient descent and variants requires careful
parameter tuning and provides no guarantee to achieve the global optimum. In
contrast we show under quite weak assumptions on the data that a particular
class of feedforward neural networks can be trained globally optimal with a
linear convergence rate with our nonlinear spectral method. Up to our knowledge
this is the first practically feasible method which achieves such a guarantee.
While the method can in principle be applied to deep networks, we restrict
ourselves for simplicity in this paper to one and two hidden layer networks.
Our experiments confirm that these models are rich enough to achieve good
performance on a series of real-world datasets.
