  As deep learning continues to advance and is applied to increasingly complex
scenarios, the demand for concurrent deployment of multiple neural network
models has arisen. This demand, commonly referred to as multi-tenant computing,
is becoming more and more important. However, even the most mature GPU-based
computing systems struggle to adequately address the significant heterogeneity
and complexity among concurrent models in terms of resource allocation and
runtime scheduling. And this usually results in considerable resource
utilization and throughput issues. To tackle these issues, this work proposes a
set of optimization techniques that advance the granularity of computing
management from both the spatial and temporal perspectives, specifically
tailored to heterogeneous model compositions for deep learning inference and
training. These techniques are further integrated as GACER -- an automated
optimization framework that provides high-utilization, high-throughput, and
low-latency multi-tenant computing support. And our experiments demonstrate
that GACER significantly improves the overall resource utilization and
consistently achieves outstanding speedups compared to native GPU computing
frameworks and existing state-of-the-art optimization works.
