  Because of the recent trends in Deep Neural Networks (DNN) models being
memory-bound, inter-operator pipelining for DNN accelerators is emerging as a
promising optimization. Inter-operator pipelining reduces costly on-chip global
memory and off-chip memory accesses by forwarding the output of a layer as the
input of the next layer within the compute array, which is proven to be an
effective optimization by previous works.
  However, the design space of inter-operator pipelining is huge, and the space
is not yet fully explored. In particular, identifying the right depth and
granularity of pipelining (or no pipelining at all) is significantly dependent
on the layer shapes and data volumes of weights and activations, and these are
different even within a domain.
  Moreover, works divide the substrate into large chunks and map one layer onto
each chunk, which requires communicating halfway through or through the global
buffer. However, for fine-grained inter-operation pipelining, placing the
corresponding consumer of the next layer tile close to the producer tile of the
current layer is a better way to exploit fine-grained spatial reuse.
  In order to support variable number of layers (ie the right depth) and
support multiple spatial organizations of layers (in accordance with the
pipelining granularity) on the substrate, we propose PipeOrgan, a new class of
spatial data organization strategy for energy efficient and congestion-free
communication between the PEs for various pipeline depth and granularity.
PipeOrgan takes advantage of flexible spatial organization and can allocate
layers to PEs based on the granularity of pipelining. We also propose changes
to the conventional mesh topology to improve the performance of coarse-grained
allocation. PipeOrgan achieves 1.95x performance improvement over the
state-of-the-art pipelined dataflow on XR-bench workloads.
