  Automatic art analysis has been mostly focused on classifying artworks into
different artistic styles. However, understanding an artistic representation
involves more complex processes, such as identifying the elements in the scene
or recognizing author influences. We present SemArt, a multi-modal dataset for
semantic art understanding. SemArt is a collection of fine-art painting images
in which each image is associated to a number of attributes and a textual
artistic comment, such as those that appear in art catalogues or museum
collections. To evaluate semantic art understanding, we envisage the Text2Art
challenge, a multi-modal retrieval task where relevant paintings are retrieved
according to an artistic text, and vice versa. We also propose several models
for encoding visual and textual artistic representations into a common semantic
space. Our best approach is able to find the correct image within the top 10
ranked images in the 45.5% of the test samples. Moreover, our models show
remarkable levels of art understanding when compared against human evaluation.
