  Human-annotated labels are often prone to noise, and the presence of such
noise will degrade the performance of the resulting deep neural network (DNN)
models. Much of the literature (with several recent exceptions) of learning
with noisy labels focuses on the case when the label noise is independent of
features. Practically, annotations errors tend to be instance-dependent and
often depend on the difficulty levels of recognizing a certain task. Applying
existing results from instance-independent settings would require a significant
amount of estimation of noise rates. Therefore, providing theoretically
rigorous solutions for learning with instance-dependent label noise remains a
challenge. In this paper, we propose CORES$^{2}$ (COnfidence REgularized Sample
Sieve), which progressively sieves out corrupted examples. The implementation
of CORES$^{2}$ does not require specifying noise rates and yet we are able to
provide theoretical guarantees of CORES$^{2}$ in filtering out the corrupted
examples. This high-quality sample sieve allows us to treat clean examples and
the corrupted ones separately in training a DNN solution, and such a separation
is shown to be advantageous in the instance-dependent noise setting. We
demonstrate the performance of CORES$^{2}$ on CIFAR10 and CIFAR100 datasets
with synthetic instance-dependent label noise and Clothing1M with real-world
human noise. As of independent interests, our sample sieve provides a generic
machinery for anatomizing noisy datasets and provides a flexible interface for
various robust training techniques to further improve the performance. Code is
available at https://github.com/UCSC-REAL/cores.
