  Structural re-parameterization has drawn increasing attention in various
computer vision tasks. It aims at improving the performance of deep models
without introducing any inference-time cost. Though efficient during inference,
such models rely heavily on the complicated training-time blocks to achieve
high accuracy, leading to large extra training cost. In this paper, we present
online convolutional re-parameterization (OREPA), a two-stage pipeline, aiming
to reduce the huge training overhead by squeezing the complex training-time
block into a single convolution. To achieve this goal, we introduce a linear
scaling layer for better optimizing the online blocks. Assisted with the
reduced training cost, we also explore some more effective re-param components.
Compared with the state-of-the-art re-param models, OREPA is able to save the
training-time memory cost by about 70% and accelerate the training speed by
around 2x. Meanwhile, equipped with OREPA, the models outperform previous
methods on ImageNet by up to +0.6%.We also conduct experiments on object
detection and semantic segmentation and show consistent improvements on the
downstream tasks. Codes are available at
https://github.com/JUGGHM/OREPA_CVPR2022 .
