  With contrastive pre-training, sentence encoders are generally optimized to
locate semantically similar samples closer to each other in their embedding
spaces. In this work, we focus on the potential of their embedding spaces to be
readily adapted to zero-shot text classification, as semantically distinct
samples are already well-separated. Our framework, RaLP (Retrieval augmented
Label Prompts for sentence encoder), encodes prompted label candidates with a
sentence encoder, then assigns the label whose prompt embedding has the highest
similarity with the input text embedding. In order to compensate for the
potentially poorly descriptive labels in their original format, RaLP retrieves
sentences that are semantically similar to the original label prompt from
external corpora and use them as additional pseudo-label prompts. RaLP achieves
competitive or stronger performance than much larger baselines on various
closed-set classification and multiple-choice QA datasets under zero-shot
settings. We show that the retrieval component plays a pivotal role in RaLP's
success, and its results are robustly attained regardless of verbalizer
variations.
