  Deep Learning optimization involves minimizing a high-dimensional loss
function in the weight space which is often perceived as difficult due to its
inherent difficulties such as saddle points, local minima, ill-conditioning of
the Hessian and limited compute resources. In this paper, we provide a
comprehensive review of $14$ standard optimization methods successfully used in
deep learning research and a theoretical assessment of the difficulties in
numerical optimization from the optimization literature.
