  We provide novel guaranteed approaches for training feedforward neural
networks with sparse connectivity. We leverage on the techniques developed
previously for learning linear networks and show that they can also be
effectively adopted to learn non-linear networks. We operate on the moments
involving label and the score function of the input, and show that their
factorization provably yields the weight matrix of the first layer of a deep
network under mild conditions. In practice, the output of our method can be
employed as effective initializers for gradient descent.
