  Few-shot classification has made great strides due to foundation models that,
through priming and prompting, are highly effective few-shot learners. However,
this approach has high variance both across different sets of few shots (data
selection) and across different finetuning runs (run variability). This is
problematic not only because it impedes the fair comparison of different
approaches, but especially because it makes few-shot learning too unreliable
for many real-world applications. To alleviate these issues, we make two
contributions for more stable and effective few-shot learning: First, we
propose novel ensembling methods and show that they substantially reduce run
variability. Second, we introduce a new active learning (AL) criterion for data
selection and present the first AL-based approach specifically tailored towards
prompt-based learning. In our experiments, we show that our combined method,
MEAL (Multiprompt finetuning and prediction Ensembling with Active Learning),
improves overall performance of prompt-based finetuning by 2.3 points on five
diverse tasks. We publicly share our code and data splits in
https://github.com/akoksal/MEAL.
