  Currently, learning better unsupervised sentence representations is the
pursuit of many natural language processing communities. Lots of approaches
based on pre-trained language models (PLMs) and contrastive learning have
achieved promising results on this task. Experimentally, we observe that the
over-smoothing problem reduces the capacity of these powerful PLMs, leading to
sub-optimal sentence representations. In this paper, we present a Simple method
named Self-Contrastive Learning (SSCL) to alleviate this issue, which samples
negatives from PLMs intermediate layers, improving the quality of the sentence
representation. Our proposed method is quite simple and can be easily extended
to various state-of-the-art models for performance boosting, which can be seen
as a plug-and-play contrastive framework for learning unsupervised sentence
representation. Extensive results prove that SSCL brings the superior
performance improvements of different strong baselines (e.g., BERT and SimCSE)
on Semantic Textual Similarity and Transfer datasets. Our codes are available
at https://github.com/nuochenpku/SSCL.
