  In recent years, even though Stochastic Gradient Descent (SGD) and its
variants are well-known for training neural networks, it suffers from
limitations such as the lack of theoretical guarantees, vanishing gradients,
and excessive sensitivity to input. To overcome these drawbacks, alternating
minimization methods have attracted fast-increasing attention recently. As an
emerging and open domain, however, several new challenges need to be addressed,
including 1) Convergence properties are sensitive to penalty parameters, and 2)
Slow theoretical convergence rate. We, therefore, propose a novel monotonous
Deep Learning Alternating Minimization (mDLAM) algorithm to deal with these two
challenges. Our innovative inequality-constrained formulation infinitely
approximates the original problem with non-convex equality constraints,
enabling our convergence proof of the proposed mDLAM algorithm regardless of
the choice of hyperparameters. Our mDLAM algorithm is shown to achieve a fast
linear convergence by the Nesterov acceleration technique. Extensive
experiments on multiple benchmark datasets demonstrate the convergence,
effectiveness, and efficiency of the proposed mDLAM algorithm. Our code is
available at \url{https://github.com/xianggebenben/mDLAM}.
