  Robustness to natural distribution shifts has seen remarkable progress thanks
to recent pre-training strategies combined with better fine-tuning methods.
However, such fine-tuning assumes access to large amounts of labelled data, and
the extent to which the observations hold when the amount of training data is
not as high remains unknown. We address this gap by performing the first
in-depth study of robustness to various natural distribution shifts in
different low-shot regimes: spanning datasets, architectures, pre-trained
initializations, and state-of-the-art robustness interventions. Most
importantly, we find that there is no single model of choice that is often more
robust than others, and existing interventions can fail to improve robustness
on some datasets even if they do so in the full-shot regime. We hope that our
work will motivate the community to focus on this problem of practical
importance.
