  Cost of serving large language models (LLM) is high, but the expensive and
scarce GPUs are poorly efficient when generating tokens sequentially, unless
the batch of sequences is enlarged. However, the batch size is limited by some
constantly reused intermediate results, namely KV-Cache. They occupy too much
memory to fit more sequences into a GPU simultaneously. While they could be
offloaded to host memory, the CPU-GPU bandwidth is an inevitable bottleneck.
  We find a way to decompose the transformer models into two parts of different
characteristics, one of which includes the memory-bound KV-Cache accessing. Our
key insight is that the aggregated memory capacity, bandwidth, and computing
power of CPUs across multiple nodes is an efficient option to process this
part. Performance improvement comes from reduced data transmission overhead and
boosted GPU throughput to process the other model part. Moreover, we address
efficiency challenges brought by heterogeneity at both temporal and
inter-device scopes using scheduling and performance modeling techniques.
Evaluation results show that our system achieves 1.88x - 5.04x the throughput
of vLLM when serving modern LLMs with the same GPU.
