  We present scalable hybrid-parallel algorithms for training large-scale 3D
convolutional neural networks. Deep learning-based emerging scientific
workflows often require model training with large, high-dimensional samples,
which can make training much more costly and even infeasible due to excessive
memory usage. We solve these challenges by extensively applying hybrid
parallelism throughout the end-to-end training pipeline, including both
computations and I/O. Our hybrid-parallel algorithm extends the standard data
parallelism with spatial parallelism, which partitions a single sample in the
spatial domain, realizing strong scaling beyond the mini-batch dimension with a
larger aggregated memory capacity. We evaluate our proposed training algorithms
with two challenging 3D CNNs, CosmoFlow and 3D U-Net. Our comprehensive
performance studies show that good weak and strong scaling can be achieved for
both networks using up 2K GPUs. More importantly, we enable training of
CosmoFlow with much larger samples than previously possible, realizing an
order-of-magnitude improvement in prediction accuracy.
