  In this paper, we investigate the popular deep learning optimization routine,
Adam, from the perspective of statistical moments. While Adam is an adaptive
lower-order moment based (of the stochastic gradient) method, we propose an
extension namely, HAdam, which uses higher order moments of the stochastic
gradient. Our analysis and experiments reveal that certain higher-order moments
of the stochastic gradient are able to achieve better performance compared to
the vanilla Adam algorithm. We also provide some analysis of HAdam related to
odd and even moments to explain some intriguing and seemingly non-intuitive
empirical results.
