  We study the problem of building text classifiers with little or no training
data, commonly known as zero and few-shot text classification. In recent years,
an approach based on neural textual entailment models has been found to give
strong results on a diverse range of tasks. In this work, we show that with
proper pre-training, Siamese Networks that embed texts and labels offer a
competitive alternative. These models allow for a large reduction in inference
cost: constant in the number of labels rather than linear. Furthermore, we
introduce label tuning, a simple and computationally efficient approach that
allows to adapt the models in a few-shot setup by only changing the label
embeddings. While giving lower performance than model fine-tuning, this
approach has the architectural advantage that a single encoder can be shared by
many different tasks.
