  Continuous learning (CL) has emerged as one of the most popular deep learning
paradigms deployed in modern cloud GPUs. Specifically, CL has the capability to
continuously update the model parameters (through model retraining) and use the
updated model (if available) to serve overtime arriving inference requests. It
is generally beneficial to co-locate the retraining and inference together to
enable timely model updates and avoid model transfer overheads. This brings the
need for GPU sharing among retraining and inferences. Meanwhile, multiple CL
workloads can share the modern GPUs in the cloud, leading to multi-tenancy
execution. In this paper, we observe that prior GPU-sharing techniques are not
optimized for multi-tenancy CL workloads. Specifically, they do not coherently
consider the accuracy of the retraining model and the inference service level
objective (SLO) attainment. Moreover, they cannot accommodate the overtime
dynamics (e.g., inference arrival intensity) in CL execution. In this paper, we
propose MIGRator, a novel GPU reconfiguration runtime that dynamically performs
GPU reconfiguration for multi-tenancy CL workloads. MIGRator is based on the
recent NVIDIA multi-instance GPU (MIG) to mitigate resource contention and
formulates the reconfiguration optimization into Integer Linear Programming
(ILP) to dynamically identify, reconfigure, and allocate the GPU instances.
MIGRator leverages the "Goodput" metric in the ILP objective function to
consider both inference SLO attainment and model accuracy in the
reconfiguration exploration. We evaluate MIGRator using representative
multi-tenancy CL workloads. The results show our approach outperforms the
state-of-the-art GPU sharing techniques (i.e., Ekya, Astraea, and PARIS) by
17\%, 21\%, and 20\%, respectively.
