  This paper aims to answer the question: Can deep learning models be
cost-efficiently trained on a global market of spot VMs spanning different data
centers and cloud providers? To provide guidance, we extensively evaluate the
cost and throughput implications of training in different zones, continents,
and clouds for representative CV, NLP, and ASR models. To expand the current
training options further, we compare the scalability potential for hybrid-cloud
scenarios by adding cloud resources to on-premise hardware to improve training
throughput. Finally, we show how leveraging spot instance pricing enables a new
cost-efficient way to train models with multiple cheap VMs, trumping both more
centralized and powerful hardware and even on-demand cloud offerings at
competitive prices.
