  This paper analyzes the pre-trained hidden representations learned from
reviews on BERT for tasks in aspect-based sentiment analysis (ABSA). Our work
is motivated by the recent progress in BERT-based language models for ABSA.
However, it is not clear how the general proxy task of (masked) language model
trained on unlabeled corpus without annotations of aspects or opinions can
provide important features for downstream tasks in ABSA. By leveraging the
annotated datasets in ABSA, we investigate both the attentions and the learned
representations of BERT pre-trained on reviews. We found that BERT uses very
few self-attention heads to encode context words (such as prepositions or
pronouns that indicating an aspect) and opinion words for an aspect. Most
features in the representation of an aspect are dedicated to the fine-grained
semantics of the domain (or product category) and the aspect itself, instead of
carrying summarized opinions from its context. We hope this investigation can
help future research in improving self-supervised learning, unsupervised
learning and fine-tuning for ABSA. The pre-trained model and code can be found
at https://github.com/howardhsu/BERT-for-RRC-ABSA.
