  This article provides a comprehensive understanding of optimization in deep
learning, with a primary focus on the challenges of gradient vanishing and
gradient exploding, which normally lead to diminished model representational
ability and training instability, respectively. We analyze these two challenges
through several strategic measures, including the improvement of gradient flow
and the imposition of constraints on a network's Lipschitz constant. To help
understand the current optimization methodologies, we categorize them into two
classes: explicit optimization and implicit optimization. Explicit optimization
methods involve direct manipulation of optimizer parameters, including weight,
gradient, learning rate, and weight decay. Implicit optimization methods, by
contrast, focus on improving the overall landscape of a network by enhancing
its modules, such as residual shortcuts, normalization methods, attention
mechanisms, and activations. In this article, we provide an in-depth analysis
of these two optimization classes and undertake a thorough examination of the
Jacobian matrices and the Lipschitz constants of many widely used deep learning
modules, highlighting existing issues as well as potential improvements.
Moreover, we also conduct a series of analytical experiments to substantiate
our theoretical discussions. This article does not aim to propose a new
optimizer or network. Rather, our intention is to present a comprehensive
understanding of optimization in deep learning. We hope that this article will
assist readers in gaining a deeper insight in this field and encourages the
development of more robust, efficient, and high-performing models.
