  Training a deep neural network (DNN) requires substantial computational and
memory requirements. It is common to use multiple devices to train a DNN to
reduce the overall training time. There are several choices to parallelize each
layer in a DNN. Exhaustively searching this list to find an optimal
parallelization strategy is prohibitively time consuming and impractical. The
standard practice is to use data parallelism because of its simplicity.
However, data parallelism is often sub-optimal, and suffers from poor
performance and high memory requirement. Expert-designed strategies have been
proposed on a case-by-case basis using domain specific knowledge. These
expert-designed strategies do not generalize well to DNNs other than the ones
for which they were designed, and are not always necessarily the best choice.
  In this paper, we propose an approach to automatically find efficient
parallelization strategies for DNNs from their computation graphs. We present
an efficient algorithm to compute these strategies within a reasonable time in
practice. We evaluate the effectiveness of our approach on various DNNs. We
also compare the performance of the strategies identified by our approach
against data parallelism, expert-designed strategies, and the state-of-the-art
approaches. Our results show that the strategies found using our approach
outperform the baseline data parallelism strategy in all the cases. In
addition, our strategies achieve better performance than the expert-designed
strategies and the state-of-the-art approaches.
