  Parameter-Efficient Fine-tuning (PEFT) facilitates the fine-tuning of Large
Language Models (LLMs) under limited resources. However, the fine-tuning
performance with PEFT on complex, knowledge-intensive tasks is limited due to
the constrained model capacity, which originates from the limited number of
additional trainable parameters. To overcome this limitation, we introduce a
novel mechanism that fine-tunes LLMs with adapters of larger size yet
memory-efficient. This is achieved by leveraging the inherent activation
sparsity in the Feed-Forward Networks (FFNs) of LLMs and utilizing the larger
capacity of Central Processing Unit (CPU) memory compared to Graphics
Processing Unit (GPU). We store and update the parameters of larger adapters on
the CPU. Moreover, we employ a Mixture of Experts (MoE)-like architecture to
mitigate unnecessary CPU computations and reduce the communication volume
between the GPU and CPU. This is particularly beneficial over the limited
bandwidth of PCI Express (PCIe). Our method can achieve fine-tuning results
comparable to those obtained with larger memory capacities, even when operating
under more limited resources such as a 24GB memory single GPU setup, with
acceptable loss in training efficiency. Our codes are available at
https://github.com/CURRENTF/MEFT.
