  Sentence embeddings encode sentences in fixed dense vectors and have played
an important role in various NLP tasks and systems. Methods for building
sentence embeddings include unsupervised learning such as Quick-Thoughts and
supervised learning such as InferSent. With the success of pretrained NLP
models, recent research shows that fine-tuning pretrained BERT on SNLI and
Multi-NLI data creates state-of-the-art sentence embeddings, outperforming
previous sentence embeddings methods on various evaluation benchmarks. In this
paper, we propose a new method to build sentence embeddings by doing supervised
contrastive learning. Specifically our method fine-tunes pretrained BERT on
SNLI data, incorporating both supervised crossentropy loss and supervised
contrastive loss. Compared with baseline where fine-tuning is only done with
supervised cross-entropy loss similar to current state-of-the-art method SBERT,
our supervised contrastive method improves 2.8% in average on Semantic Textual
Similarity (STS) benchmarks and 1.05% in average on various sentence transfer
tasks.
