  Convolutional layers are the core building blocks of Convolutional Neural
Networks (CNNs). In this paper, we propose to augment a convolutional layer
with an additional depthwise convolution, where each input channel is convolved
with a different 2D kernel. The composition of the two convolutions constitutes
an over-parameterization, since it adds learnable parameters, while the
resulting linear operation can be expressed by a single convolution layer. We
refer to this depthwise over-parameterized convolutional layer as DO-Conv. We
show with extensive experiments that the mere replacement of conventional
convolutional layers with DO-Conv layers boosts the performance of CNNs on many
classical vision tasks, such as image classification, detection, and
segmentation. Moreover, in the inference phase, the depthwise convolution is
folded into the conventional convolution, reducing the computation to be
exactly equivalent to that of a convolutional layer without
over-parameterization. As DO-Conv introduces performance gains without
incurring any computational complexity increase for inference, we advocate it
as an alternative to the conventional convolutional layer. We open-source a
reference implementation of DO-Conv in Tensorflow, PyTorch and GluonCV at
https://github.com/yangyanli/DO-Conv.
