  In this paper, we propose Self-Contrastive Decorrelation (SCD), a
self-supervised approach. Given an input sentence, it optimizes a joint
self-contrastive and decorrelation objective. Learning a representation is
facilitated by leveraging the contrast arising from the instantiation of
standard dropout at different rates. The proposed method is conceptually simple
yet empirically powerful. It achieves comparable results with state-of-the-art
methods on multiple benchmarks without using contrastive pairs. This study
opens up avenues for efficient self-supervised learning methods that are more
robust than current contrastive methods.
