  With the increasing popularity of robotics in industrial control and
autonomous driving, deep reinforcement learning (DRL) raises the attention of
various fields. However, DRL computation on the modern powerful GPU platform is
still inefficient due to its heterogeneous workloads and interleaved execution
paradigm. To this end, we propose GMI-DRL, a systematic design to accelerate
multi-GPU DRL via GPU spatial multiplexing. We introduce a novel design of
resource-adjustable GPU multiplexing instances (GMIs) to match the actual needs
of DRL tasks, an adaptive GMI management strategy to simultaneously achieve
high GPU utilization and computation throughput, and a highly efficient
inter-GMI communication support to meet the demands of various DRL
communication patterns. Comprehensive experiments reveal that GMI-DRL
outperforms state-of-the-art NVIDIA Isaac Gym with NCCL (up to 2.81X) and
Horovod (up to 2.34X) support in training throughput on the latest DGX-A100
platform. Our work provides an initial user experience with GPU spatial
multiplexing in processing heterogeneous workloads with a mixture of
computation and communication.
