  Residual block is a very common component in recent state-of-the art CNNs
such as EfficientNet or EfficientDet. Shortcut data accounts for nearly 40% of
feature-maps access in ResNet152 [8]. Most of the previous DNN compilers,
accelerators ignore the shortcut data optimization. This paper presents
ShortcutFusion, an optimization tool for FPGA-based accelerator with a
reuse-aware static memory allocation for shortcut data, to maximize on-chip
data reuse given resource constraints. From TensorFlow DNN models, the proposed
design generates instruction sets for a group of nodes which uses an optimized
data reuse for each residual block. The accelerator design implemented on the
Xilinx KCU1500 FPGA card 2.8x faster and 9.9x more power efficient than NVIDIA
RTX 2080 Ti for 256x256 input size. . Compared to the result from baseline, in
which the weights, inputs, and outputs are accessed from the off-chip memory
exactly once per each layer, ShortcutFusion reduces the DRAM access by
47.8-84.8% for RetinaNet, Yolov3, ResNet152, and EfficientNet. Given a similar
buffer size to ShortcutMining [8], which also mine the shortcut data in
hardware, the proposed work reduces off-chip access for feature-maps 5.27x
while accessing weight from off-chip memory exactly once.
