  In recent years, self-supervised learning has played a pivotal role in
advancing machine learning by allowing models to acquire meaningful
representations from unlabeled data. An intriguing research avenue involves
developing self-supervised models within an information-theoretic framework,
but many studies often deviate from the stochasticity assumptions made when
deriving their objectives. To gain deeper insights into this issue, we propose
to explicitly model the representation with stochastic embeddings and assess
their effects on performance, information compression and potential for
out-of-distribution detection. From an information-theoretic perspective, we
seek to investigate the impact of probabilistic modeling on the information
bottleneck, shedding light on a trade-off between compression and preservation
of information in both representation and loss space. Emphasizing the
importance of distinguishing between these two spaces, we demonstrate how
constraining one can affect the other, potentially leading to performance
degradation. Moreover, our findings suggest that introducing an additional
bottleneck in the loss space can significantly enhance the ability to detect
out-of-distribution examples, only leveraging either representation features or
the variance of their underlying distribution.
