  Although residual connection enables training very deep neural networks, it
is not friendly for online inference due to its multi-branch topology. This
encourages many researchers to work on designing DNNs without residual
connections at inference. For example, RepVGG re-parameterizes multi-branch
topology to a VGG-like (single-branch) model when deploying, showing great
performance when the network is relatively shallow. However, RepVGG can not
transform ResNet to VGG equivalently because re-parameterizing methods can only
be applied to linear blocks and the non-linear layers (ReLU) have to be put
outside of the residual connection which results in limited representation
ability, especially for deeper networks. In this paper, we aim to remedy this
problem and propose to remove the residual connection in a vanilla ResNet
equivalently by a reserving and merging (RM) operation on ResBlock.
Specifically, the RM operation allows input feature maps to pass through the
block while reserving their information and merges all the information at the
end of each block, which can remove residual connections without changing the
original output. As a plug-in method, RM Operation basically has three
advantages: 1) its implementation makes it naturally friendly for high ratio
network pruning. 2) it helps break the depth limitation of RepVGG. 3) it leads
to better accuracy-speed trade-off network (RMNet) compared to ResNet and
RepVGG. We believe the ideology of RM Operation can inspire many insights on
model design for the community in the future. Code is available at:
https://github.com/fxmeng/RMNet.
