  Among the three main components (data, labels, and models) of any supervised
learning system, data and models have been the main subjects of active
research. However, studying labels and their properties has received very
little attention. Current principles and paradigms of labeling impose several
challenges to machine learning algorithms. Labels are often incomplete,
ambiguous, and redundant. In this paper we study the effects of various
properties of labels and introduce the Label Refinery: an iterative procedure
that updates the ground truth labels after examining the entire dataset. We
show significant gain using refined labels across a wide range of models. Using
a Label Refinery improves the state-of-the-art top-1 accuracy of (1) AlexNet
from 59.3 to 67.2, (2) MobileNet from 70.6 to 73.39, (3) MobileNet-0.25 from
50.6 to 55.59, (4) VGG19 from 72.7 to 75.46, and (5) Darknet19 from 72.9 to
74.47.
