  Convolutional neural networks are witnessing wide adoption in computer vision
systems with numerous applications across a range of visual recognition tasks.
Much of this progress is fueled through advances in convolutional neural
network architectures and learning algorithms even as the basic premise of a
convolutional layer has remained unchanged. In this paper, we seek to revisit
the convolutional layer that has been the workhorse of state-of-the-art visual
recognition models. We introduce a very simple, yet effective, module called a
perturbation layer as an alternative to a convolutional layer. The perturbation
layer does away with convolution in the traditional sense and instead computes
its response as a weighted linear combination of non-linearly activated
additive noise perturbed inputs. We demonstrate both analytically and
empirically that this perturbation layer can be an effective replacement for a
standard convolutional layer. Empirically, deep neural networks with
perturbation layers, called Perturbative Neural Networks (PNNs), in lieu of
convolutional layers perform comparably with standard CNNs on a range of visual
datasets (MNIST, CIFAR-10, PASCAL VOC, and ImageNet) with fewer parameters.
