  Deep neural networks (DNN) use a wide range of network topologies to achieve
high accuracy within diverse applications. This model diversity makes it
impossible to identify a single "dataflow" (execution schedule) to perform
optimally across all possible layers and network topologies. Several frameworks
support the exploration of the best dataflow for a given DNN layer and
hardware. However, switching the dataflow from one layer to the next layer
within one DNN model can result in hardware inefficiencies stemming from memory
data layout mismatch among the layers. Unfortunately, all existing frameworks
treat each layer independently and typically model memories as black boxes (one
large monolithic wide memory), which ignores the data layout and can not deal
with the data layout dependencies of sequential layers. These frameworks are
not capable of doing dataflow cross-layer optimization. This work, hence, aims
at cross-layer dataflow optimization, taking the data dependency and data
layout reshuffling overheads among layers into account. Additionally, we
propose to exploit the multibank memories typically present in modern DNN
accelerators towards efficiently reshuffling data to support more dataflow at
low overhead. These innovations are supported through the Cross-layer
Memory-aware Dataflow Scheduler (CMDS). CMDS can model DNN execution
energy/latency while considering the different data layout requirements due to
the varied optimal dataflow of layers. Compared with the state-of-the-art
(SOTA), which performs layer-optimized memory-unaware scheduling, CMDS achieves
up to 5.5X energy reduction and 1.35X latency reduction with negligible
hardware cost.
