  This work provides an additional step in the theoretical understanding of
neural networks. We consider neural networks with one hidden layer and show
that when learning symmetric functions, one can choose initial conditions so
that standard SGD training efficiently produces generalization guarantees. We
empirically verify this and show that this does not hold when the initial
conditions are chosen at random. The proof of convergence investigates the
interaction between the two layers of the network. Our results highlight the
importance of using symmetry in the design of neural networks.
