  Deep neural networks have incredible capacity and expressibility, and can
seemingly memorize any training set. This introduces a problem when training in
the presence of noisy labels, as the noisy examples cannot be distinguished
from clean examples by the end of training. Recent research has dealt with this
challenge by utilizing the fact that deep networks seem to memorize clean
examples much earlier than noisy examples. Here we report a new empirical
result: for each example, when looking at the time it has been memorized by
each model in an ensemble of networks, the diversity seen in noisy examples is
much larger than the clean examples. We use this observation to develop a new
method for noisy labels filtration. The method is based on a statistics of the
data, which captures the differences in ensemble learning dynamics between
clean and noisy data. We test our method on three tasks: (i) noise amount
estimation; (ii) noise filtration; (iii) supervised classification. We show
that our method improves over existing baselines in all three tasks using a
variety of datasets, noise models, and noise levels. Aside from its improved
performance, our method has two other advantages. (i) Simplicity, which implies
that no additional hyperparameters are introduced. (ii) Our method is modular:
it does not work in an end-to-end fashion, and can therefore be used to clean a
dataset for any other future usage.
