  Single image-level annotations only correctly describe an often small subset
of an image's content, particularly when complex real-world scenes are
depicted. While this might be acceptable in many classification scenarios, it
poses a significant challenge for applications where the set of classes differs
significantly between training and test time. In this paper, we take a closer
look at the implications in the context of $\textit{few-shot learning}$.
Splitting the input samples into patches and encoding these via the help of
Vision Transformers allows us to establish semantic correspondences between
local regions across images and independent of their respective class. The most
informative patch embeddings for the task at hand are then determined as a
function of the support set via online optimization at inference time,
additionally providing visual interpretability of `$\textit{what matters
most}$' in the image. We build on recent advances in unsupervised training of
networks via masked image modelling to overcome the lack of fine-grained labels
and learn the more general statistical structure of the data while avoiding
negative image-level annotation influence, $\textit{aka}$ supervision collapse.
Experimental results show the competitiveness of our approach, achieving new
state-of-the-art results on four popular few-shot classification benchmarks for
$5$-shot and $1$-shot scenarios.
