  As emerging deep neural network (DNN) models continue to grow in size, using
large GPU clusters to train DNNs is becoming an essential requirement to
achieving acceptable training times. In this paper, we consider the case where
future increases in cluster size will cause the global batch size that can be
used to train models to reach a fundamental limit: beyond a certain point,
larger global batch sizes cause sample efficiency to degrade, increasing
overall time to accuracy. As a result, to achieve further improvements in
training performance, we must instead consider "strong scaling" strategies that
hold the global batch size constant and allocate smaller batches to each GPU.
Unfortunately, this makes it significantly more difficult to use cluster
resources efficiently. We present DeepPool, a system that addresses this
efficiency challenge through two key ideas. First, burst parallelism allocates
large numbers of GPUs to foreground jobs in bursts to exploit the unevenness in
parallelism across layers. Second, GPU multiplexing prioritizes throughput for
foreground training jobs, while packing in background training jobs to reclaim
underutilized GPU resources, thereby improving cluster-wide utilization.
Together, these two ideas enable DeepPool to deliver a 1.2 - 2.3x improvement
in total cluster throughput over standard data parallelism with a single task
when the cluster scale is large.
