  Recently, sentiment analysis has seen remarkable advance with the help of
pre-training approaches. However, sentiment knowledge, such as sentiment words
and aspect-sentiment pairs, is ignored in the process of pre-training, despite
the fact that they are widely used in traditional sentiment analysis
approaches. In this paper, we introduce Sentiment Knowledge Enhanced
Pre-training (SKEP) in order to learn a unified sentiment representation for
multiple sentiment analysis tasks. With the help of automatically-mined
knowledge, SKEP conducts sentiment masking and constructs three sentiment
knowledge prediction objectives, so as to embed sentiment information at the
word, polarity and aspect level into pre-trained sentiment representation. In
particular, the prediction of aspect-sentiment pairs is converted into
multi-label classification, aiming to capture the dependency between words in a
pair. Experiments on three kinds of sentiment tasks show that SKEP
significantly outperforms strong pre-training baseline, and achieves new
state-of-the-art results on most of the test datasets. We release our code at
https://github.com/baidu/Senta.
