  Large language models (LLMs) are increasingly integrated into many online
services, yet they remain cost-prohibitive to deploy due to the requirement of
expensive GPU instances. Prior work has addressed the high cost of LLM serving
by improving the inference engine, but less attention has been given to
selecting the most cost-efficient GPU type(s) for a specific LLM service. There
is a large and growing landscape of GPU types and, within these options, higher
cost does not always lead to increased performance. Instead, through a
comprehensive investigation, we find that three key LLM service characteristics
(request size, request rate, SLO) strongly influence GPU cost efficiency, and
differing GPU types are most cost efficient for differing LLM service settings.
As a result, the most cost-efficient allocation for a given service is
typically a mix of heterogeneous GPU types. Based on this analysis, we
introduce M\'elange, a GPU allocation framework that navigates these diverse
LLM service characteristics and heterogeneous GPU option space to automatically
and efficiently derive the minimal-cost GPU allocation for a given LLM service.
We formulate the GPU allocation task as a cost-aware bin packing problem where
GPUs are bins and items are slices of the service workload. Our formulation's
constraints account for a service's unique characteristics, allowing M\'elange
to be flexible to support diverse service settings and heterogeneity-aware to
adapt the GPU allocation to a specific service. Compared to using only a single
GPU type, M\'elange reduces deployment costs by up to 77% in conversational
settings, 33% in document-based settings, and 51% in a mixed setting.
