  With the rapid growth in the number of large language model (LLM) users, it
is difficult for bandwidth-constrained cloud servers to simultaneously process
massive LLM services in real-time. Recently, edge-cloud infrastructures have
been used to improve the processing efficiency of large-scale LLM services.
However, the diversity of task requirements and the dynamics of resources pose
great challenges to inference scheduling, leading to the wastage of many
resources. In this paper, we present PerLLM, a personalized inference
scheduling framework with edge-cloud collaboration designed for diverse LLM
services. For the complexity of multiple constraints and the decision-making
process of edge-cloud collaboration, we integrate the upper confidence bound
algorithm based on the constraint satisfaction mechanism in PerLLM. For diverse
LLM services, PerLLM can optimize service scheduling and resource allocation
solutions within the edge-cloud infrastructure to meet processing time
requirements while minimizing energy costs. Experimental results from different
model deployments show that PerLLM can effectively meet the processing time
requirements of personalized services. Compared to other methods, PerLLM
achieves 2.2x, 2.1x, and 1.6x throughput and reduces the energy cost by more
than 50%.
