  This paper aims to provide understandings for the effect of an
over-parameterized model, e.g. a deep neural network, memorizing
instance-dependent noisy labels. We first quantify the harms caused by
memorizing noisy instances, and show the disparate impacts of noisy labels for
sample instances with different representation frequencies. We then analyze how
several popular solutions for learning with noisy labels mitigate this harm at
the instance level. Our analysis reveals that existing approaches lead to
disparate treatments when handling noisy instances. While higher-frequency
instances often enjoy a high probability of an improvement by applying these
solutions, lower-frequency instances do not. Our analysis reveals new
understandings for when these approaches work, and provides theoretical
justifications for previously reported empirical observations. This observation
requires us to rethink the distribution of label noise across instances and
calls for different treatments for instances in different regimes.
