  Recently, Supervised Contrastive Learning (SCL) has been shown to achieve
excellent performance in most classification tasks. In SCL, a neural network is
trained to optimize two objectives: pull an anchor and positive samples
together in the embedding space, and push the anchor apart from the negatives.
However, these two different objectives may conflict, requiring trade-offs
between them during optimization. In this work, we formulate the SCL problem as
a Multi-Objective Optimization problem for the fine-tuning phase of RoBERTa
language model. Two methods are utilized to solve the optimization problem: (i)
the linear scalarization (LS) method, which minimizes a weighted linear
combination of pertask losses; and (ii) the Exact Pareto Optimal (EPO) method
which finds the intersection of the Pareto front with a given preference
vector. We evaluate our approach on several GLUE benchmark tasks, without using
data augmentations, memory banks, or generating adversarial examples. The
empirical results show that the proposed learning strategy significantly
outperforms a strong competitive contrastive learning baseline
