  A painter is free to modify how components of a natural scene are depicted,
which can lead to a perceptually convincing image of the distal world. This
signals a major difference between photos and paintings: paintings are
explicitly created for human perception. Studying these painterly depictions
could be beneficial to a multidisciplinary audience. In this paper, we capture
and explore the painterly depictions of materials to enable the study of
depiction and perception of materials through the artists' eye. We annotated a
dataset of 19k paintings with 200k+ bounding boxes from which polygon segments
were automatically extracted. Each bounding box was assigned a coarse label
(e.g., fabric) and a fine-grained label (e.g., velvety, silky). We demonstrate
the cross-disciplinary utility of our dataset by presenting novel findings
across art history, human perception, and computer vision. Our experiments
include analyzing the distribution of materials depicted in paintings, showing
how painters create convincing depictions using a stylized approach, and
demonstrating how paintings can be used to build more robust computer vision
models. We conclude that our dataset of painterly material depictions is a rich
source for gaining insights into the depiction and perception of materials
across multiple disciplines. The MIP dataset is freely accessible at
https://materialsinpaintings.tudelft.nl
