  Neural methods for SA have led to quantitative improvements over previous
approaches, but these advances are not always accompanied with a thorough
analysis of the qualitative differences. Therefore, it is not clear what
outstanding conceptual challenges for sentiment analysis remain. In this work,
we attempt to discover what challenges still prove a problem for sentiment
classifiers for English and to provide a challenging dataset. We collect the
subset of sentences that an (oracle) ensemble of state-of-the-art sentiment
classifiers misclassify and then annotate them for 18 linguistic and
paralinguistic phenomena, such as negation, sarcasm, modality, etc. The dataset
is available at https://github.com/ltgoslo/assessing_and_probing_sentiment.
Finally, we provide a case study that demonstrates the usefulness of the
dataset to probe the performance of a given sentiment classifier with respect
to linguistic phenomena.
