  Today, ground-truth generation uses data sets annotated by cloud-based
annotation services. These services rely on human annotation, which can be
prohibitively expensive. In this paper, we consider the problem of hybrid
human-machine labeling, which trains a classifier to accurately auto-label part
of the data set. However, training the classifier can be expensive too. We
propose an iterative approach that minimizes total overall cost by, at each
step, jointly determining which samples to label using humans and which to
label using the trained classifier. We validate our approach on well known
public data sets such as Fashion-MNIST, CIFAR-10, CIFAR-100, and ImageNet. In
some cases, our approach has 6x lower overall cost relative to human labeling
the entire data set, and is always cheaper than the cheapest competing
strategy.
