  Large pre-trained models can dramatically reduce the amount of task-specific
data required to solve a problem, but they often fail to capture
domain-specific nuances out of the box. The Web likely contains the information
necessary to excel on any specific application, but identifying the right data
a priori is challenging. This paper shows how to leverage recent advances in
NLP and multi-modal learning to augment a pre-trained model with search engine
retrieval. We propose to retrieve useful data from the Web at test time based
on test cases that the model is uncertain about. Different from existing
retrieval-augmented approaches, we then update the model to address this
underlying uncertainty. We demonstrate substantial improvements in zero-shot
performance, e.g. a remarkable increase of 15 percentage points in accuracy on
the Stanford Cars and Flowers datasets. We also present extensive experiments
that explore the impact of noisy retrieval and different learning strategies.
