  This paper establishes a mathematical foundation for the Adam optimizer,
elucidating its connection to natural gradient descent through Riemannian and
information geometry. We provide an accessible and detailed analysis of the
diagonal empirical Fisher information matrix (FIM) in Adam, clarifying all
detailed approximations and advocating for the use of log probability functions
as loss, which should be based on discrete distributions, due to the
limitations of empirical FIM. Our analysis uncovers flaws in the original Adam
algorithm, leading to proposed corrections such as enhanced momentum
calculations, adjusted bias corrections, adaptive epsilon, and gradient
clipping. We refine the weight decay term based on our theoretical framework.
Our modified algorithm, Fisher Adam (FAdam), demonstrates superior performance
across diverse domains including LLM, ASR, and VQ-VAE, achieving
state-of-the-art results in ASR.
