  Quasi-Newton methods still face significant challenges in training
large-scale neural networks due to additional compute costs in the Hessian
related computations and instability issues in stochastic training. A
well-known method, L-BFGS that efficiently approximates the Hessian using
history parameter and gradient changes, suffers convergence instability in
stochastic training. So far, attempts that adapt L-BFGS to large-scale
stochastic training incur considerable extra overhead, which offsets its
convergence benefits in wall-clock time. In this paper, we propose mL-BFGS, a
lightweight momentum-based L-BFGS algorithm that paves the way for quasi-Newton
(QN) methods in large-scale distributed deep neural network (DNN) optimization.
mL-BFGS introduces a nearly cost-free momentum scheme into L-BFGS update and
greatly reduces stochastic noise in the Hessian, therefore stabilizing
convergence during stochastic optimization. For model training at a large
scale, mL-BFGS approximates a block-wise Hessian, thus enabling distributing
compute and memory costs across all computing nodes. We provide a supporting
convergence analysis for mL-BFGS in stochastic settings. To investigate mL-BFGS
potential in large-scale DNN training, we train benchmark neural models using
mL-BFGS and compare performance with baselines (SGD, Adam, and other
quasi-Newton methods). Results show that mL-BFGS achieves both noticeable
iteration-wise and wall-clock speedup.
