  Learning high-quality sentence representations benefits a wide range of
natural language processing tasks. Though BERT-based pre-trained language
models achieve high performance on many downstream tasks, the native derived
sentence representations are proved to be collapsed and thus produce a poor
performance on the semantic textual similarity (STS) tasks. In this paper, we
present ConSERT, a Contrastive Framework for Self-Supervised Sentence
Representation Transfer, that adopts contrastive learning to fine-tune BERT in
an unsupervised and effective way. By making use of unlabeled texts, ConSERT
solves the collapse issue of BERT-derived sentence representations and make
them more applicable for downstream tasks. Experiments on STS datasets
demonstrate that ConSERT achieves an 8\% relative improvement over the previous
state-of-the-art, even comparable to the supervised SBERT-NLI. And when further
incorporating NLI supervision, we achieve new state-of-the-art performance on
STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples
available, showing its robustness in data scarcity scenarios.
