  Modern deep neural network (DNN) trainings utilize various training
techniques, e.g., nonlinear activation functions, batch normalization,
skip-connections, etc. Despite their effectiveness, it is still mysterious how
they help accelerate DNN trainings in practice. In this paper, we provide an
empirical study of the regularization effect of these training techniques on
DNN optimization. Specifically, we find that the optimization trajectories of
successful DNN trainings consistently obey a certain regularity principle that
regularizes the model update direction to be aligned with the trajectory
direction. Theoretically, we show that such a regularity principle leads to a
convergence guarantee in nonconvex optimization and the convergence rate
depends on a regularization parameter. Empirically, we find that DNN trainings
that apply the training techniques achieve a fast convergence and obey the
regularity principle with a large regularization parameter, implying that the
model updates are well aligned with the trajectory. On the other hand, DNN
trainings without the training techniques have slow convergence and obey the
regularity principle with a small regularization parameter, implying that the
model updates are not well aligned with the trajectory. Therefore, different
training techniques regularize the model update direction via the regularity
principle to facilitate the convergence.
