  Convolutional neural networks are among the most successful architectures in
deep learning with this success at least partially attributable to the efficacy
of spatial invariance as an inductive bias. Locally connected layers, which
differ from convolutional layers only in their lack of spatial invariance,
usually perform poorly in practice. However, these observations still leave
open the possibility that some degree of relaxation of spatial invariance may
yield a better inductive bias than either convolution or local connectivity. To
test this hypothesis, we design a method to relax the spatial invariance of a
network layer in a controlled manner; we create a \textit{low-rank} locally
connected layer, where the filter bank applied at each position is constructed
as a linear combination of basis set of filter banks with spatially varying
combining weights. By varying the number of basis filter banks, we can control
the degree of relaxation of spatial invariance. In experiments with small
convolutional networks, we find that relaxing spatial invariance improves
classification accuracy over both convolution and locally connected layers
across MNIST, CIFAR-10, and CelebA datasets, thus suggesting that spatial
invariance may be an overly restrictive prior.
