  Transfer learning enables the sharing of common knowledge among models for a
variety of downstream tasks, but traditional methods suffer in limited training
data settings and produce narrow models incapable of effectively generalizing
under distribution shifts. Foundation models have recently demonstrated
impressive zero-shot inference capabilities and robustness under distribution
shifts. However, zero-shot evaluation for these models has been predominantly
confined to benchmarks with simple distribution shifts, limiting our
understanding of their effectiveness under the more realistic shifts found in
practice. Moreover, common fine-tuning methods for these models have yet to be
evaluated against vision models in few-shot scenarios where training data is
limited. To address these gaps, we present a new recipe for few-shot
fine-tuning of the popular vision-language foundation model CLIP and evaluate
its performance on challenging benchmark datasets with realistic distribution
shifts from the WILDS collection. Our experimentation demonstrates that, while
zero-shot CLIP fails to match performance of trained vision models on more
complex benchmarks, few-shot CLIP fine-tuning outperforms its vision-only
counterparts in terms of in-distribution and out-of-distribution accuracy at
all levels of training data availability. This provides a strong incentive for
adoption of foundation models within few-shot learning applications operating
with real-world data. Code is available at
https://github.com/mit-ll/robust-vision-language-finetuning
