  In this work, we perform a wide variety of experiments with different deep
learning architectures on datasets of limited size. According to our study, we
show that model complexity is a critical factor when only a few samples per
class are available. Differently from the literature, we show that in some
configurations, the state of the art can be improved using low complexity
models. For instance, in problems with scarce training samples and without data
augmentation, low-complexity convolutional neural networks perform comparably
well or better than state-of-the-art architectures. Moreover, we show that even
standard data augmentation can boost recognition performance by large margins.
This result suggests the development of more complex data
generation/augmentation pipelines for cases when data is limited. Finally, we
show that dropout, a widely used regularization technique, maintains its role
as a good regularizer even when data is scarce. Our findings are empirically
validated on the sub-sampled versions of popular CIFAR-10, Fashion-MNIST and,
SVHN benchmarks.
