  Recent literature has found that an effective method to customize or further
improve large language models (LLMs) is to add dynamic adapters, such as
low-rank adapters (LoRA) with Mixture-of-Experts (MoE) structures. Though such
dynamic adapters incur modest computational complexity, they surprisingly lead
to huge inference latency overhead, slowing down the decoding speed by 2.5+
times. In this paper, we analyze the fine-grained costs of the dynamic adapters
and find that the fragmented CUDA kernel calls are the root cause. Therefore,
we propose LoRA-Switch, a system-algorithm co-designed architecture for
efficient dynamic adapters. Unlike most existing dynamic structures that adopt
layer-wise or block-wise dynamic routing, LoRA-Switch introduces a token-wise
routing mechanism. It switches the LoRA adapters and weights for each token and
merges them into the backbone for inference. For efficiency, this switching is
implemented with an optimized CUDA kernel, which fuses the merging operations
for all LoRA adapters at once. Based on experiments with popular open-source
LLMs on common benchmarks, our approach has demonstrated similar accuracy
improvement as existing dynamic adapters, while reducing the decoding latency
by more than 2.4 times.
