  In this study, we introduce a novel adaptive optimizer, H-Fac, which
incorporates a factorized approach to momentum and scaling parameters. Our
algorithm demonstrates competitive performances on both ResNets and Vision
Transformers, while achieving sublinear memory costs through the use of rank-1
parameterizations for moment estimators. We develop our algorithms based on
principles derived from Hamiltonian dynamics, providing robust theoretical
underpinnings. These optimization algorithms are designed to be both
straightforward and adaptable, facilitating easy implementation in diverse
settings.
