  This work presents BAdam, an optimization method that leverages the block
coordinate descent framework with Adam as the inner solver. BAdam offers a
memory efficient approach to the full parameter finetuning of large language
models. We conduct theoretical convergence analysis for BAdam in the
deterministic case. Experimentally, we apply BAdam to instruction-tune the
Llama 2-7B and Llama 3-8B models using a single RTX3090-24GB GPU. The results
confirm BAdam's efficiency in terms of memory and running time. Additionally,
the convergence verification indicates that BAdam exhibits superior convergence
behavior compared to LoRA. Furthermore, the downstream performance evaluation
using the MT-bench shows that BAdam modestly surpasses LoRA and more
substantially outperforms LOMO. Finally, we compare BAdam with Adam on a
medium-sized task, i.e., finetuning RoBERTa-large on the SuperGLUE benchmark.
The results demonstrate that BAdam is capable of narrowing the performance gap
with Adam more effectively than LoRA. Our code is available at
https://github.com/Ledzy/BAdam.
