  Given new tasks with very little data$-$such as new classes in a
classification problem or a domain shift in the input$-$performance of modern
vision systems degrades remarkably quickly. In this work, we illustrate how the
neural network representations which underpin modern vision systems are subject
to supervision collapse, whereby they lose any information that is not
necessary for performing the training task, including information that may be
necessary for transfer to new tasks or domains. We then propose two methods to
mitigate this problem. First, we employ self-supervised learning to encourage
general-purpose features that transfer better. Second, we propose a novel
Transformer based neural network architecture called CrossTransformers, which
can take a small number of labeled images and an unlabeled query, find coarse
spatial correspondence between the query and the labeled images, and then infer
class membership by computing distances between spatially-corresponding
features. The result is a classifier that is more robust to task and domain
shift, which we demonstrate via state-of-the-art performance on Meta-Dataset, a
recent dataset for evaluating transfer from ImageNet to many other vision
datasets.
