  Recently, sentiment-aware pre-trained language models (PLMs) demonstrate
impressive results in downstream sentiment analysis tasks. However, they
neglect to evaluate the quality of their constructed sentiment representations;
they just focus on improving the fine-tuning performance, which overshadows the
representation quality. We argue that without guaranteeing the representation
quality, their downstream performance can be highly dependent on the
supervision of the fine-tuning data rather than representation quality. This
problem would make them difficult to foray into other sentiment-related
domains, especially where labeled data is scarce. We first propose
Sentiment-guided Textual Similarity (SgTS), a novel metric for evaluating the
quality of sentiment representations, which is designed based on the degree of
equivalence in sentiment polarity between two sentences. We then propose
SentiCSE, a novel Sentiment-aware Contrastive Sentence Embedding framework for
constructing sentiment representations via combined word-level and
sentence-level objectives, whose quality is guaranteed by SgTS. Qualitative and
quantitative comparison with the previous sentiment-aware PLMs shows the
superiority of our work. Our code is available at:
https://github.com/nayohan/SentiCSE
