  Due to the lack of large-scale datasets, the prevailing approach in visual
sentiment analysis is to leverage models trained for object classification in
large datasets like ImageNet. However, objects are sentiment neutral which
hinders the expected gain of transfer learning for such tasks. In this work, we
propose to overcome this problem by learning a novel sentiment-aligned image
embedding that is better suited for subsequent visual sentiment analysis. Our
embedding leverages the intricate relation between emojis and images in
large-scale and readily available data from social media. Emojis are
language-agnostic, consistent, and carry a clear sentiment signal which make
them an excellent proxy to learn a sentiment aligned embedding. Hence, we
construct a novel dataset of 4 million images collected from Twitter with their
associated emojis. We train a deep neural model for image embedding using emoji
prediction task as a proxy. Our evaluation demonstrates that the proposed
embedding outperforms the popular object-based counterpart consistently across
several sentiment analysis benchmarks. Furthermore, without bell and whistles,
our compact, effective and simple embedding outperforms the more elaborate and
customized state-of-the-art deep models on these public benchmarks.
Additionally, we introduce a novel emoji representation based on their visual
emotional response which supports a deeper understanding of the emoji modality
and their usage on social media.
