  The use of contrastive loss for representation learning has become prominent
in computer vision, and it is now getting attention in Natural Language
Processing (NLP). Here, we explore the idea of using a batch-softmax
contrastive loss when fine-tuning large-scale pre-trained transformer models to
learn better task-specific sentence embeddings for pairwise sentence scoring
tasks. We introduce and study a number of variations in the calculation of the
loss as well as in the overall training procedure; in particular, we find that
data shuffling can be quite important. Our experimental results show sizable
improvements on a number of datasets and pairwise sentence scoring tasks
including classification, ranking, and regression. Finally, we offer detailed
analysis and discussion, which should be useful for researchers aiming to
explore the utility of contrastive loss in NLP.
