  Serving generative inference of the large language model is a crucial
component of contemporary AI applications. This paper focuses on deploying such
services in a heterogeneous and cross-datacenter setting to mitigate the
substantial inference costs typically associated with a single centralized
datacenter. Towards this end, we propose HexGen, a flexible distributed
inference engine that uniquely supports the asymmetric partition of generative
inference computations over both tensor model parallelism and pipeline
parallelism and allows for effective deployment across diverse GPUs
interconnected by a fully heterogeneous network. We further propose a
sophisticated scheduling algorithm grounded in constrained optimization that
can adaptively assign asymmetric inference computation across the GPUs to
fulfill inference requests while maintaining acceptable latency levels. We
conduct an extensive evaluation to verify the efficiency of HexGen by serving
the state-of-the-art Llama-2 (70B) model. The results suggest that HexGen can
choose to achieve up to 2.3 times lower latency deadlines or tolerate up to 4
times more request rates compared with the homogeneous baseline given the same
budget.
