  With the rise of artificial intelligence in recent years, Deep Neural
Networks (DNNs) have been widely used in many domains. To achieve high
performance and energy efficiency, hardware acceleration (especially inference)
of DNNs is intensively studied both in academia and industry. However, we still
face two challenges: large DNN models and datasets, which incur frequent
off-chip memory accesses; and the training of DNNs, which is not well-explored
in recent accelerator designs. To truly provide high throughput and energy
efficient acceleration for the training of deep and large models, we inevitably
need to use multiple accelerators to explore the coarse-grain parallelism,
compared to the fine-grain parallelism inside a layer considered in most of the
existing architectures. It poses the key research question to seek the best
organization of computation and dataflow among accelerators. In this paper,
inspired by recent work in machine learning systems, we propose a solution
HyPar to determine layer-wise parallelism for deep neural network training with
an array of DNN accelerators. HyPar partitions the feature map tensors (input
and output), the kernel tensors, the gradient tensors, and the error tensors
for the DNN accelerators. A partition constitutes the choice of parallelism for
weighted layers. The optimization target is to search a partition that
minimizes the total communication during training a complete DNN. To solve this
problem, we propose a communication model to explain the source and amount of
communications. Then, we use a hierarchical layer-wise dynamic programming
method to search for the partition for each layer.
