  Building large models with parameter sharing accounts for most of the success
of deep convolutional neural networks (CNNs). In this paper, we propose doubly
convolutional neural networks (DCNNs), which significantly improve the
performance of CNNs by further exploring this idea. In stead of allocating a
set of convolutional filters that are independently learned, a DCNN maintains
groups of filters where filters within each group are translated versions of
each other. Practically, a DCNN can be easily implemented by a two-step
convolution procedure, which is supported by most modern deep learning
libraries. We perform extensive experiments on three image classification
benchmarks: CIFAR-10, CIFAR-100 and ImageNet, and show that DCNNs consistently
outperform other competing architectures. We have also verified that replacing
a convolutional layer with a doubly convolutional layer at any depth of a CNN
can improve its performance. Moreover, various design choices of DCNNs are
demonstrated, which shows that DCNN can serve the dual purpose of building more
accurate models and/or reducing the memory footprint without sacrificing the
accuracy.
