  Zero-shot text classifiers based on label descriptions embed an input text
and a set of labels into the same space: measures such as cosine similarity can
then be used to select the most similar label description to the input text as
the predicted label. In a true zero-shot setup, designing good label
descriptions is challenging because no development set is available. Inspired
by the literature on Learning with Disagreements, we look at how probabilistic
models of repeated rating analysis can be used for selecting the best label
descriptions in an unsupervised fashion. We evaluate our method on a set of
diverse datasets and tasks (sentiment, topic and stance). Furthermore, we show
that multiple, noisy label descriptions can be aggregated to boost the
performance.
