  Solving image classification tasks given small training datasets remains an
open challenge for modern computer vision. Aggressive data augmentation and
generative models are among the most straightforward approaches to overcoming
the lack of data. However, the first fails to be agnostic to varying image
domains, while the latter requires additional compute and careful design. In
this work, we study alternative regularization strategies to push the limits of
supervised learning on small image classification datasets. In particular,
along with the model size and training schedule scaling, we employ a heuristic
to select (semi) optimal learning rate and weight decay couples via the norm of
model parameters. By training on only 1% of the original CIFAR-10 training set
(i.e., 50 images per class) and testing on ciFAIR-10, a variant of the original
CIFAR without duplicated images, we reach a test accuracy of 66.5%, on par with
the best state-of-the-art methods.
