  Pretrained language models have improved zero-shot text classification by
allowing the transfer of semantic knowledge from the training data in order to
classify among specific label sets in downstream tasks. We propose a simple way
to further improve zero-shot accuracies with minimal effort. We curate small
finetuning datasets intended to describe the labels for a task. Unlike typical
finetuning data, which has texts annotated with labels, our data simply
describes the labels in language, e.g., using a few related terms,
dictionary/encyclopedia entries, and short templates. Across a range of topic
and sentiment datasets, our method is more accurate than zero-shot by 17-19%
absolute. It is also more robust to choices required for zero-shot
classification, such as patterns for prompting the model to classify and
mappings from labels to tokens in the model's vocabulary. Furthermore, since
our data merely describes the labels but does not use input texts, finetuning
on it yields a model that performs strongly on multiple text domains for a
given label set, even improving over few-shot out-of-domain classification in
multiple settings.
