  A particularly successful class of approaches for few-shot learning combines
language models with prompts -- hand-crafted task descriptions that complement
data samples. However, designing prompts by hand for each task commonly
requires domain knowledge and substantial guesswork. We observe, in the context
of classification tasks, that instruction finetuned language models exhibit
remarkable prompt robustness, and we subsequently propose a simple method to
eliminate the need for handcrafted prompts, named AuT-Few. This approach
consists of (i) a prompt retrieval module that selects suitable task
instructions from the instruction-tuning knowledge base, and (ii) the
generation of two distinct, semantically meaningful, class descriptions and a
selection mechanism via cross-validation. Over $12$ datasets, spanning $8$
classification tasks, we show that AuT-Few outperforms current state-of-the-art
few-shot learning methods. Moreover, AuT-Few is the best ranking method across
datasets on the RAFT few-shot benchmark. Notably, these results are achieved
without task-specific handcrafted prompts on unseen tasks.
