  We introduce Train/Test-Time Adaptation with Retrieval (${\rm T^3AR}$), a
method to adapt models both at train and test time by means of a retrieval
module and a searchable pool of external samples. Before inference, ${\rm
T^3AR}$ adapts a given model to the downstream task using refined pseudo-labels
and a self-supervised contrastive objective function whose noise distribution
leverages retrieved real samples to improve feature adaptation on the target
data manifold. The retrieval of real images is key to ${\rm T^3AR}$ since it
does not rely solely on synthetic data augmentations to compensate for the lack
of adaptation data, as typically done by other adaptation algorithms.
Furthermore, thanks to the retrieval module, our method gives the user or
service provider the possibility to improve model adaptation on the downstream
task by incorporating further relevant data or to fully remove samples that may
no longer be available due to changes in user preference after deployment.
First, we show that ${\rm T^3AR}$ can be used at training time to improve
downstream fine-grained classification over standard fine-tuning baselines, and
the fewer the adaptation data the higher the relative improvement (up to 13%).
Second, we apply ${\rm T^3AR}$ for test-time adaptation and show that
exploiting a pool of external images at test-time leads to more robust
representations over existing methods on DomainNet-126 and VISDA-C, especially
when few adaptation data are available (up to 8%).
