  Spatio-Temporal Convolutional Neural Networks (ST-CNN) allow extending CNN
capabilities from image processing to consecutive temporal-pattern recognition.
Generally, state-of-the-art (SotA) ST-CNNs inflate the feature maps and weights
from well-known CNN backbones to represent the additional time dimension.
However, edge computing applications would suffer tremendously from such large
computation or memory overhead. Fortunately, the overlapping nature of ST-CNN
enables various optimizations, such as the dilated causal convolution structure
and Depth-First (DF) layer fusion to reuse the computation between time steps
and CNN sliding windows, respectively. Yet, no hardware-aware approach has been
proposed that jointly explores the optimal strategy from a scheduling as well
as a hardware point of view. To this end, we present ACCO, an automated
optimizer that explores efficient Causal CNN transformation and DF scheduling
for ST-CNNs on edge hardware accelerators. By cost-modeling the computation and
data movement on the accelerator architecture, ACCO automatically selects the
best scheduling strategy for the given hardware-algorithm target. Compared to
the fixed dilated causal structure, ST-CNNs with ACCO reach an ~8.4x better
Energy-Delay-Product. Meanwhile, ACCO improves ~20% in layer-fusion optimals
compared to the SotA DF exploration toolchain. When jointly optimizing ST-CNN
on the temporal and spatial dimension, ACCO's scheduling outcomes are on
average 19x faster and 37x more energy-efficient than spatial DF schemes.
