  Deep convolutional neural networks have achieved remarkable progress in
recent years. However, the large volume of intermediate results generated
during inference poses a significant challenge to the accelerator design for
resource-constraint FPGA. Due to the limited on-chip storage, partial results
of intermediate layers are frequently transferred back and forth between
on-chip memory and off-chip DRAM, leading to a non-negligible increase in
latency and energy consumption. In this paper, we propose block convolution, a
hardware-friendly, simple, yet efficient convolution operation that can
completely avoid the off-chip transfer of intermediate feature maps at
run-time. The fundamental idea of block convolution is to eliminate the
dependency of feature map tiles in the spatial dimension when spatial tiling is
used, which is realized by splitting a feature map into independent blocks so
that convolution can be performed separately on individual blocks. We conduct
extensive experiments to demonstrate the efficacy of the proposed block
convolution on both the algorithm side and the hardware side. Specifically, we
evaluate block convolution on 1) VGG-16, ResNet-18, ResNet-50, and MobileNet-V1
for ImageNet classification task; 2) SSD, FPN for COCO object detection task,
and 3) VDSR for Set5 single image super-resolution task. Experimental results
demonstrate that comparable or higher accuracy can be achieved with block
convolution. We also showcase two CNN accelerators via algorithm/hardware
co-design based on block convolution on memory-limited FPGAs, and evaluation
shows that both accelerators substantially outperform the baseline without
off-chip transfer of intermediate feature maps.
