  Many state-of-the-art Deep Neural Networks (DNNs) have substantial memory
requirements. Limited device memory becomes a bottleneck when training those
models. We propose ParDNN, an automatic, generic, and non-intrusive
partitioning strategy for DNNs that are represented as computational graphs.
ParDNN decides a placement of DNN's underlying computational graph operations
across multiple devices so that the devices' memory constraints are met and the
training time is minimized. ParDNN is completely independent of the deep
learning aspects of a DNN. It requires no modification neither at the model nor
at the systems level implementation of its operation kernels. ParDNN partitions
DNNs having billions of parameters and hundreds of thousands of operations in
seconds to few minutes. Our experiments with TensorFlow on 16 GPUs demonstrate
efficient training of 5 very large models while achieving superlinear scaling
for both the batch size and training throughput. ParDNN either outperforms or
qualitatively improves upon the related work.
