  Driven by the tremendous effort in researching novel deep learning (DL)
algorithms, the training cost of developing new models increases staggeringly
in recent years. We analyze GPU cluster usage statistics from a top research
institute for more insights into the hardware efficiency achieved by typical DL
training jobs. Our study reveals that single-accelerator training jobs can
dominate the cluster-wide resource consumption when launched repetitively
(e.g., for hyper-parameter tuning) while severely under-utilizing the hardware.
Fortunately, we observe that such workloads have the following unique
characteristics: (i) the models among jobs often have the same types of
operators with the same shapes, and (ii) the inter-model horizontal fusion of
such operators is mathematically equivalent to other already well-optimized
operators. Thus, to help DL researchers and practitioners effectively improve
the hardware utilization of their novel DL training workloads, we propose
Horizontally Fused Training Array (HFTA). HFTA is a new DL framework extension
library that horizontally fuses the models from different repetitive jobs
deeply down to operators and then trains them simultaneously on a shared
accelerator. To show the generality of our solution, we apply HFTA to six DL
models training on state-of-the-art accelerators (GPUs and TPUs). Our results
indicate that HFTA is highly effective in improving hardware utilization and
achieves up to $15.1 \times$ higher training throughput vs. the standard
practice of running each job on a separate accelerator.
