  Contrastive approaches to representation learning have recently shown great
promise. In contrast to generative approaches, these contrastive models learn a
deterministic encoder with no notion of uncertainty or confidence. In this
paper, we introduce a simple approach based on "contrasting distributions" that
learns to assign uncertainty for pretrained contrastive representations. In
particular, we train a deep network from a representation to a distribution in
representation space, whose variance can be used as a measure of confidence. In
our experiments, we show that this deep uncertainty model can be used (1) to
visually interpret model behavior, (2) to detect new noise in the input to
deployed models, (3) to detect anomalies, where we outperform 10 baseline
methods across 11 tasks with improvements of up to 14% absolute, and (4) to
classify out-of-distribution examples where our fully unsupervised model is
competitive with supervised methods.
