  Recent progress on few-shot learning largely relies on annotated data for
meta-learning: base classes sampled from the same domain as the novel classes.
However, in many applications, collecting data for meta-learning is infeasible
or impossible. This leads to the cross-domain few-shot learning problem, where
there is a large shift between base and novel class domains. While
investigations of the cross-domain few-shot scenario exist, these works are
limited to natural images that still contain a high degree of visual
similarity. No work yet exists that examines few-shot learning across different
imaging methods seen in real world scenarios, such as aerial and medical
imaging. In this paper, we propose the Broader Study of Cross-Domain Few-Shot
Learning (BSCD-FSL) benchmark, consisting of image data from a diverse
assortment of image acquisition methods. This includes natural images, such as
crop disease images, but additionally those that present with an increasing
dissimilarity to natural images, such as satellite images, dermatology images,
and radiology images. Extensive experiments on the proposed benchmark are
performed to evaluate state-of-art meta-learning approaches, transfer learning
approaches, and newer methods for cross-domain few-shot learning. The results
demonstrate that state-of-art meta-learning methods are surprisingly
outperformed by earlier meta-learning approaches, and all meta-learning methods
underperform in relation to simple fine-tuning by 12.8% average accuracy.
Performance gains previously observed with methods specialized for cross-domain
few-shot learning vanish in this more challenging benchmark. Finally, accuracy
of all methods tend to correlate with dataset similarity to natural images,
verifying the value of the benchmark to better represent the diversity of data
seen in practice and guiding future research.
