  Despite the surprising few-shot performance of in-context learning (ICL), it
is still a common practice to randomly sample examples to serve as context.
This paper advocates a new principle for ICL: self-adaptive in-context
learning. The self-adaption mechanism is introduced to help each sample find an
in-context example permutation (i.e., selection and ordering) that can derive
the correct prediction, thus maximizing performance. To validate the
effectiveness of self-adaptive ICL, we propose a general select-then-rank
framework and instantiate it with new selection and ranking algorithms. Upon
extensive evaluation on eight different NLP datasets, our self-adaptive ICL
method achieves a 40% relative improvement over the common practice setting.
Further analysis reveals the enormous potential of self-adaptive ICL that it
might be able to close the gap between ICL and finetuning given more advanced
algorithms. Our code is released to facilitate future research in this area:
https://github.com/Shark-NLP/self-adaptive-ICL
