  In cloud machine learning (ML) inference systems, providing low latency to
end-users is of utmost importance. However, maximizing server utilization and
system throughput is also crucial for ML service providers as it helps lower
the total-cost-of-ownership. GPUs have oftentimes been criticized for ML
inference usages as its massive compute and memory throughput is hard to be
fully utilized under low-batch inference scenarios. To address such limitation,
NVIDIA's recently announced Ampere GPU architecture provides features to
"reconfigure" one large, monolithic GPU into multiple smaller "GPU partitions".
Such feature provides cloud ML service providers the ability to utilize the
reconfigurable GPU not only for large-batch training but also for small-batch
inference with the potential to achieve high resource utilization. In this
paper, we study this emerging GPU architecture with reconfigurability to
develop a high-performance multi-GPU ML inference server. Our first proposition
is a sophisticated partitioning algorithm for reconfigurable GPUs that
systematically determines a heterogeneous set of multi-granular GPU partitions,
best suited for the inference server's deployment. Furthermore, we co-design an
elastic scheduling algorithm tailored for our heterogeneously partitioned GPU
server which effectively balances low latency and high GPU utilization.
