  In this paper, we propose a geometric framework to analyze the convergence
properties of gradient descent trajectories in the context of linear neural
networks. We translate a well-known empirical observation of linear neural nets
into a conjecture that we call the \emph{overfitting conjecture} which states
that, for almost all training data and initial conditions, the trajectory of
the corresponding gradient descent system converges to a global minimum. This
would imply that the solution achieved by vanilla gradient descent algorithms
is equivalent to that of the least-squares estimation, for linear neural
networks of an arbitrary number of hidden layers. Built upon a key invariance
property induced by the network structure, we first establish convergence of
gradient descent trajectories to critical points of the square loss function in
the case of linear networks of arbitrary depth. Our second result is the proof
of the \emph{overfitting conjecture} in the case of single-hidden-layer linear
networks with an argument based on the notion of normal hyperbolicity and under
a generic property on the training data (i.e., holding for almost all training
data).
