  Existing research on learning with noisy labels mainly focuses on synthetic
label noise. Synthetic noise, though has clean structures which greatly enabled
statistical analyses, often fails to model real-world noise patterns. The
recent literature has observed several efforts to offer real-world noisy
datasets, yet the existing efforts suffer from two caveats: (1) The lack of
ground-truth verification makes it hard to theoretically study the property and
treatment of real-world label noise; (2) These efforts are often of large
scales, which may result in unfair comparisons of robust methods within
reasonable and accessible computation power. To better understand real-world
label noise, it is crucial to build controllable and moderate-sized real-world
noisy datasets with both ground-truth and noisy labels. This work presents two
new benchmark datasets CIFAR-10N, CIFAR-100N, equipping the training datasets
of CIFAR-10, CIFAR-100 with human-annotated real-world noisy labels we
collected from Amazon Mechanical Turk. We quantitatively and qualitatively show
that real-world noisy labels follow an instance-dependent pattern rather than
the classically assumed and adopted ones (e.g., class-dependent label noise).
We then initiate an effort to benchmarking a subset of the existing solutions
using CIFAR-10N and CIFAR-100N. We further proceed to study the memorization of
correct and wrong predictions, which further illustrates the difference between
human noise and class-dependent synthetic noise. We show indeed the real-world
noise patterns impose new and outstanding challenges as compared to synthetic
label noise. These observations require us to rethink the treatment of noisy
labels, and we hope the availability of these two datasets would facilitate the
development and evaluation of future learning with noisy label solutions.
Datasets and leaderboards are available at http://noisylabels.com.
