  Few shot in-context learning (ICL) typically assumes access to large
annotated training sets. However, in many real world scenarios, such as domain
adaptation, there is only a limited budget to annotate a small number of
samples, with the goal of maximizing downstream performance. We study various
methods for selecting samples to annotate within a predefined budget,
specifically focusing on the named entity recognition (NER) task, which has
real-world applications, is expensive to annotate, and is relatively less
studied in ICL setups. Across different models and datasets, we find that a
relatively small pool of annotated samples can achieve results comparable to
using the entire training set. Moreover, we discover that random selection of
samples for annotation yields surprisingly good performance. Finally, we
observe that a diverse annotation pool is correlated with improved performance.
We hope that future work adopts our realistic paradigm which takes annotation
budget into account.
