  Implicit regularization is an important way to interpret neural networks.
Recent theory starts to explain implicit regularization with the model of deep
matrix factorization (DMF) and analyze the trajectory of discrete gradient
dynamics in the optimization process. These discrete gradient dynamics are
relatively small but not infinitesimal, thus fitting well with the practical
implementation of neural networks. Currently, discrete gradient dynamics
analysis has been successfully applied to shallow networks but encounters the
difficulty of complex computation for deep networks. In this work, we introduce
another discrete gradient dynamics approach to explain implicit regularization,
i.e. landscape analysis. It mainly focuses on gradient regions, such as saddle
points and local minima. We theoretically establish the connection between
saddle point escaping (SPE) stages and the matrix rank in DMF. We prove that,
for a rank-R matrix reconstruction, DMF will converge to a second-order
critical point after R stages of SPE. This conclusion is further experimentally
verified on a low-rank matrix reconstruction problem. This work provides a new
theory to analyze implicit regularization in deep learning.
