  Linear networks provide valuable insights into the workings of neural
networks in general. This paper identifies conditions under which the gradient
flow provably trains a linear network, in spite of the non-strict saddle points
present in the optimization landscape. This paper also provides the
computational complexity of training linear networks with gradient flow. To
achieve these results, this work develops a machinery to provably identify the
stable set of gradient flow, which then enables us to improve over the state of
the art in the literature of linear networks (Bah et al., 2019;Arora et al.,
2018a). Crucially, our results appear to be the first to break away from the
lazy training regime which has dominated the literature of neural networks.
This work requires the network to have a layer with one neuron, which subsumes
the networks with a scalar output, but extending the results of this
theoretical work to all linear networks remains a challenging open problem.
