  The remarkable advancements in large language models (LLMs) have
significantly enhanced the performance in few-shot learning settings. By using
only a small number of labeled examples, referred to as demonstrations, LLMs
can effectively grasp the task at hand through in-context learning. However,
the process of selecting appropriate demonstrations has received limited
attention in prior work. This paper addresses the issue of identifying the most
informative demonstrations for few-shot learning by approaching it as a
pool-based Active Learning (AL) problem over a single iteration. Our objective
is to investigate how AL algorithms can serve as effective demonstration
selection methods for in-context learning. We compare various standard AL
algorithms based on uncertainty, diversity, and similarity, and consistently
observe that the latter outperforms all other methods, including random
sampling. Notably, uncertainty sampling, despite its success in conventional
supervised learning scenarios, performs poorly in this context. Our extensive
experimentation involving a diverse range of GPT and OPT models across $24$
classification and multi-choice tasks, coupled with thorough analysis,
unambiguously demonstrates that in-context example selection through AL
prioritizes high-quality examples that exhibit low uncertainty and bear
similarity to the test examples.
