  Meta-learning algorithms are widely used for few-shot learning. For example,
image recognition systems that readily adapt to unseen classes after seeing
only a few labeled examples. Despite their success, we show that modern
meta-learning algorithms are extremely sensitive to the data used for
adaptation, i.e. support data. In particular, we demonstrate the existence of
(unaltered, in-distribution, natural) images that, when used for adaptation,
yield accuracy as low as 4\% or as high as 95\% on standard few-shot image
classification benchmarks. We explain our empirical findings in terms of class
margins, which in turn suggests that robust and safe meta-learning requires
larger margins than supervised learning.
