  The impact of transformer networks is booming, yet, they come with
significant computational complexity. It is therefore essential to understand
how to optimally map and execute these networks on modern neural processor
hardware. So far, literature on transformer scheduling optimization has been
focusing on deployment on GPU and specific ASICs. This work enables extensive
hardware/mapping exploration by extending the DSE framework Stream towards
support for transformers across a wide variety of hardware architectures and
different execution schedules. After validation, we explore the optimal
schedule for transformer layers/attention heads and investigate whether layer
fusion is beneficial to improve latency, energy or memory requirements. Our
study shows that the memory requirements for active feature data can be
drastically reduced, by adapting the execution schedule based on the size of
the input of the attention head.
