  The ability to learn from limited data, or few-shot learning, is a desirable
and often critical requirement for NLP systems. While many existing methods do
poorly at learning from a handful of examples, large pretrained language models
have recently been shown to be efficient few-shot learners. One approach to
few-shot learning, which does not require finetuning of model parameters, is to
augment the language model's input with priming text which is typically
constructed using task specific descriptions and examples. In this work, we
further explore priming-based few-shot learning, with focus on using examples
as prompts. We show that presenting examples in the right order is key for
generalization. We introduce PERO (Prompting with Examples in the Right Order),
where we formulate few-shot learning as search over the set of permutations of
the training examples. We show that PERO can learn to generalize efficiently
using as few as 10 examples, in contrast to existing approaches. While the
newline token is a natural choice for separating the examples in the prompt, we
show that learning a new separator token can potentially provide further gains
in performance. We demonstrate the effectiveness of the proposed method on the
tasks of sentiment classification, natural language inference and fact
retrieval. Finally, we analyze the learned prompts to reveal novel insights,
including the idea that two training examples in the right order alone can
provide competitive performance for sentiment classification and natural
language inference.
