  Lowering the memory requirement in full-parameter training on large models
has become a hot research area. MeZO fine-tunes the large language models
(LLMs) by just forward passes in a zeroth-order SGD optimizer (ZO-SGD),
demonstrating excellent performance with the same GPU memory usage as
inference. However, the simulated perturbation stochastic approximation for
gradient estimate in MeZO leads to severe oscillations and incurs a substantial
time overhead. Moreover, without momentum regularization, MeZO shows severe
over-fitting problems. Lastly, the perturbation-irrelevant momentum on ZO-SGD
does not improve the convergence rate. This study proposes ZO-AdaMU to resolve
the above problems by adapting the simulated perturbation with momentum in its
stochastic approximation. Unlike existing adaptive momentum methods, we
relocate momentum on simulated perturbation in stochastic gradient
approximation. Our convergence analysis and experiments prove this is a better
way to improve convergence stability and rate in ZO-SGD. Extensive experiments
demonstrate that ZO-AdaMU yields better generalization for LLMs fine-tuning
across various NLP tasks than MeZO and its momentum variants.
