  Self-supervised models trained with a contrastive loss such as CLIP have
shown to be very powerful in zero-shot classification settings. However, to be
used as a zero-shot classifier these models require the user to provide new
captions over a fixed set of labels at test time. In many settings, it is hard
or impossible to know if a new query caption is compatible with the source
captions used to train the model. We address these limitations by framing the
zero-shot classification task as an outlier detection problem and develop a
conformal prediction procedure to assess when a given test caption may be
reliably used. On a real-world medical example, we show that our proposed
conformal procedure improves the reliability of CLIP-style models in the
zero-shot classification setting, and we provide an empirical analysis of the
factors that may affect its performance.
