  The typical process for developing LLMs involves pre-training a general
foundation model on massive data, followed by fine-tuning on task-specific data
to create specialized experts. Serving these experts poses challenges, as
loading all experts onto devices is impractical, and frequent switching between
experts in response to user requests incurs substantial I/O costs, increasing
latency and expenses. Previous approaches decompose expert weights into
pre-trained model weights and residual delta weights, then quantize the delta
weights to reduce model size. However, these methods often lead to significant
quantization errors at extremely low bitwidths and assume the appropriate model
for a user request is known in advance, which is not practical. To address
these issues, we introduce ME-Switch, a memory-efficient expert switching
framework for LLM serving. ME-Switch uses mixed-precision quantization,
selectively quantizing non-salient input channels of delta weights to extremely
low bits while keeping salient ones intact, significantly reducing storage
demands while maintaining performance. Additionally, we develop a routing
method that efficiently directs user queries to the most suitable expert by
transforming the model selection problem into a domain classification problem.
Extensive experiments show ME-Switch's promising memory efficiency and routing
performance. For example, when serving three models from the Mistral-7B family,
ME-Switch reduces model size by 1.74x while maintaining nearly lossless
performance on instruction, mathematical reasoning, and code generation tasks.
Furthermore, ME-Switch can efficiently serve 16 models from the Mistral-7B
family on a single NVIDIA A100 GPU.
