  The prevailing paradigm for producing semantic segmentation training data
relies on densely labelling each pixel of each image in the training set, akin
to colouring-in books. This approach becomes a bottleneck when scaling up in
the number of images, classes, and annotators. Here we propose instead a
pointillist approach for semantic segmentation annotation, where only
point-wise yes/no questions are answered. We explore design alternatives for
such an active learning approach, measure the speed and consistency of human
annotators on this task, show that this strategy enables training good
segmentation models, and that it is suitable for evaluating models at test
time. As concrete proof of the scalability of our method, we collected and
released 22.6M point labels over 4,171 classes on the Open Images dataset. Our
results enable to rethink the semantic segmentation pipeline of annotation,
training, and evaluation from a pointillism point of view.
