  Systems for training massive deep learning models (billions of parameters)
today assume and require specialized "hyper-clusters": hundreds or thousands of
GPUs wired with specialized high-bandwidth interconnects such as NV-Link and
Infiniband. Besides being expensive, such dependence on hyper-clusters and
custom high-speed inter-connects limits the size of such clusters, creating (a)
scalability limits on job parallelism; (b) resource fragmentation across
hyper-clusters.
  In this paper, we present Varuna, a new system that enables training massive
deep learning models on commodity networking. Varuna makes thrifty use of
networking resources and automatically configures the user's training job to
efficiently use any given set of resources. Therefore, Varuna is able to
leverage "low-priority" VMs that cost about 5x cheaper than dedicated GPUs,
thus significantly reducing the cost of training massive models. We demonstrate
the efficacy of Varuna by training massive models, including a 200 billion
parameter model, on 5x cheaper "spot VMs", while maintaining high training
throughput. Varuna improves end-to-end training time by up to 18x compared to
other model-parallel approaches and up to 26% compared to other pipeline
parallel approaches.
  The code for Varuna is available at https://github.com/microsoft/varuna.
