  We study the implicit regularization imposed by gradient descent for learning
multi-layer homogeneous functions including feed-forward fully connected and
convolutional deep neural networks with linear, ReLU or Leaky ReLU activation.
We rigorously prove that gradient flow (i.e. gradient descent with
infinitesimal step size) effectively enforces the differences between squared
norms across different layers to remain invariant without any explicit
regularization. This result implies that if the weights are initially small,
gradient flow automatically balances the magnitudes of all layers. Using a
discretization argument, we analyze gradient descent with positive step size
for the non-convex low-rank asymmetric matrix factorization problem without any
regularization. Inspired by our findings for gradient flow, we prove that
gradient descent with step sizes $\eta_t = O\left(t^{-\left(
\frac12+\delta\right)} \right)$ ($0<\delta\le\frac12$) automatically balances
two low-rank factors and converges to a bounded global optimum. Furthermore,
for rank-$1$ asymmetric matrix factorization we give a finer analysis showing
gradient descent with constant step size converges to the global minimum at a
globally linear rate. We believe that the idea of examining the invariance
imposed by first order algorithms in learning homogeneous models could serve as
a fundamental building block for studying optimization for learning deep
models.
