  Learning powerful feature representations with CNNs is hard when training
data are limited. Pre-training is one way to overcome this, but it requires
large datasets sufficiently similar to the target domain. Another option is to
design priors into the model, which can range from tuned hyperparameters to
fully engineered representations like Scattering Networks. We combine these
ideas into structured receptive field networks, a model which has a fixed
filter basis and yet retains the flexibility of CNNs. This flexibility is
achieved by expressing receptive fields in CNNs as a weighted sum over a fixed
basis which is similar in spirit to Scattering Networks. The key difference is
that we learn arbitrary effective filter sets from the basis rather than
modeling the filters. This approach explicitly connects classical multiscale
image analysis with general CNNs. With structured receptive field networks, we
improve considerably over unstructured CNNs for small and medium dataset
scenarios as well as over Scattering for large datasets. We validate our
findings on ILSVRC2012, Cifar-10, Cifar-100 and MNIST. As a realistic small
dataset example, we show state-of-the-art classification results on popular 3D
MRI brain-disease datasets where pre-training is difficult due to a lack of
large public datasets in a similar domain.
