  Recent works have shown that line search methods greatly increase performance
of traditional stochastic gradient descent methods on a variety of datasets and
architectures [1], [2]. In this work we succeed in extending line search
methods to the novel and highly popular Transformer architecture and dataset
domains in natural language processing. More specifically, we combine the
Armijo line search with the Adam optimizer and extend it by subdividing the
networks architecture into sensible units and perform the line search
separately on these local units. Our optimization method outperforms the
traditional Adam optimizer and achieves significant performance improvements
for small data sets or small training budgets, while performing equal or better
for other tested cases. Our work is publicly available as a python package,
which provides a hyperparameter-free pytorch optimizer that is compatible with
arbitrary network architectures.
