  Both efficient neural networks and hardware accelerators are being explored
to speed up DNN inference on edge devices. For example, MobileNet uses
depthwise separable convolution to achieve much lower latency, while systolic
arrays provide much higher performance per watt. Interestingly however, the
combination of these two ideas is inefficient: The computational patterns of
depth-wise separable convolution are not systolic and lack data reuse to
saturate the systolic array's constrained dataflow. In this paper, we propose
FuSeConv (Fully-Separable Convolution) as a drop-in replacement for depth-wise
separable convolution. FuSeConv generalizes the decomposition of convolutions
fully to separable 1D convolutions along spatial and depth dimensions. The
resultant computation is systolic and efficiently utilizes the systolic array
with a slightly modified dataflow. With FuSeConv, we achieve a significant
speed-up of 3x-7x with the MobileNet family of networks on a systolic array of
size 64x64, with comparable accuracy on the ImageNet dataset. The high speed-up
motivates exploration of hardware-aware Neural Operator Search (NOS) in
complement to ongoing efforts on Neural Architecture Search (NAS).
