  Deep learning has paved the way for strong recognition systems which are
often both trained on and applied to natural images. In this paper, we examine
the give-and-take relationship between such visual recognition systems and the
rich information available in the fine arts. First, we find that visual
recognition systems designed for natural images can work surprisingly well on
paintings. In particular, we find that interactive segmentation tools can be
used to cleanly annotate polygonal segments within paintings, a task which is
time consuming to undertake by hand. We also find that FasterRCNN, a model
which has been designed for object recognition in natural scenes, can be
quickly repurposed for detection of materials in paintings. Second, we show
that learning from paintings can be beneficial for neural networks that are
intended to be used on natural images. We find that training on paintings
instead of natural images can improve the quality of learned features and we
further find that a large number of paintings can be a valuable source of test
data for evaluating domain adaptation algorithms. Our experiments are based on
a novel large-scale annotated database of material depictions in paintings
which we detail in a separate manuscript.
