  We categorize meta-learning evaluation into two settings:
$\textit{in-distribution}$ [ID], in which the train and test tasks are sampled
$\textit{iid}$ from the same underlying task distribution, and
$\textit{out-of-distribution}$ [OOD], in which they are not. While most
meta-learning theory and some FSL applications follow the ID setting, we
identify that most existing few-shot classification benchmarks instead reflect
OOD evaluation, as they use disjoint sets of train (base) and test (novel)
classes for task generation. This discrepancy is problematic because -- as we
show on numerous benchmarks -- meta-learning methods that perform better on
existing OOD datasets may perform significantly worse in the ID setting. In
addition, in the OOD setting, even though current FSL benchmarks seem
befitting, our study highlights concerns in 1) reliably performing model
selection for a given meta-learning method, and 2) consistently comparing the
performance of different methods. To address these concerns, we provide
suggestions on how to construct FSL benchmarks to allow for ID evaluation as
well as more reliable OOD evaluation. Our work aims to inform the meta-learning
community about the importance and distinction of ID vs. OOD evaluation, as
well as the subtleties of OOD evaluation with current benchmarks.
