  Numerically locating the critical points of non-convex surfaces is a
long-standing problem central to many fields. Recently, the loss surfaces of
deep neural networks have been explored to gain insight into outstanding
questions in optimization, generalization, and network architecture design.
However, the degree to which recently-proposed methods for numerically
recovering critical points actually do so has not been thoroughly evaluated. In
this paper, we examine this issue in a case for which the ground truth is
known: the deep linear autoencoder. We investigate two sub-problems associated
with numerical critical point identification: first, because of large parameter
counts, it is infeasible to find all of the critical points for contemporary
neural networks, necessitating sampling approaches whose characteristics are
poorly understood; second, the numerical tolerance for accurately identifying a
critical point is unknown, and conservative tolerances are difficult to
satisfy. We first identify connections between recently-proposed methods and
well-understood methods in other fields, including chemical physics, economics,
and algebraic geometry. We find that several methods work well at recovering
certain information about loss surfaces, but fail to take an unbiased sample of
critical points. Furthermore, numerical tolerance must be very strict to ensure
that numerically-identified critical points have similar properties to true
analytical critical points. We also identify a recently-published Newton method
for optimization that outperforms previous methods as a critical point-finding
algorithm. We expect our results will guide future attempts to numerically
study critical points in large nonlinear neural networks.
