  Sentiment Analysis has seen much progress in the past two decades. For the
past few years, neural network approaches, primarily RNNs and CNNs, have been
the most successful for this task. Recently, a new category of neural networks,
self-attention networks (SANs), have been created which utilizes the attention
mechanism as the basic building block. Self-attention networks have been shown
to be effective for sequence modeling tasks, while having no recurrence or
convolutions. In this work we explore the effectiveness of the SANs for
sentiment analysis. We demonstrate that SANs are superior in performance to
their RNN and CNN counterparts by comparing their classification accuracy on
six datasets as well as their model characteristics such as training speed and
memory consumption. Finally, we explore the effects of various SAN
modifications such as multi-head attention as well as two methods of
incorporating sequence position information into SANs.
