  Pre-trained language models (PLMs) have been shown effective for zero-shot
(0shot) text classification. 0shot models based on natural language inference
(NLI) and next sentence prediction (NSP) employ cross-encoder architecture and
infer by making a forward pass through the model for each label-text pair
separately. This increases the computational cost to make inferences linearly
in the number of labels. In this work, we improve the efficiency of such
cross-encoder-based 0shot models by restricting the number of likely labels
using another fast base classifier-based conformal predictor (CP) calibrated on
samples labeled by the 0shot model. Since a CP generates prediction sets with
coverage guarantees, it reduces the number of target labels without excluding
the most probable label based on the 0shot model. We experiment with three
intent and two topic classification datasets. With a suitable CP for each
dataset, we reduce the average inference time for NLI- and NSP-based models by
25.6% and 22.2% respectively, without dropping performance below the predefined
error rate of 1%.
