  The scalability of large language models (LLMs) in handling high-complexity
models and large-scale datasets has led to tremendous successes in pivotal
domains. While there is an urgent need to acquire more training data for LLMs,
a concerning reality is the depletion of high-quality public datasets within a
few years. In view of this, the federated learning (FL) LLM fine-tuning
paradigm recently has been proposed to facilitate collaborative LLM fine-tuning
on distributed private data, where multiple data owners collaboratively
fine-tune a shared LLM without sharing raw data. However, the staggering model
size of LLMs imposes heavy computing and communication burdens on clients,
posing significant barriers to the democratization of the FL LLM fine-tuning
paradigm. To address this issue, split learning (SL) has emerged as a promising
solution by offloading the primary training workload to a server via model
partitioning while exchanging activation/activation's gradients with smaller
data sizes rather than the entire LLM. Unfortunately, research on the SL LLM
fine-tuning paradigm is still in its nascent stage. To fill this gap, in this
paper, we propose the first SL LLM fine-tuning framework, named SplitLoRA.
SplitLoRA is built on the split federated learning (SFL) framework,
amalgamating the advantages of parallel training from FL and model splitting
from SL and thus greatly enhancing the training efficiency. It is worth noting
that SplitLoRA is the inaugural open-source benchmark for SL LLM fine-tuning,
providing a foundation for research efforts dedicated to advancing SL LLM
fine-tuning. Extensive simulations validate that SplitLoRA achieves target
accuracy in significantly less time than state-of-the-art LLM fine-tuning
frameworks, demonstrating the superior training performance of SplitLoRA. The
project page is available at https://fduinc.github.io/splitlora/.
