  Deep neural networks (DNNs) are capable of perfectly fitting the training
data, including memorizing noisy data. It is commonly believed that
memorization hurts generalization. Therefore, many recent works propose
mitigation strategies to avoid noisy data or correct memorization. In this
work, we step back and ask the question: Can deep learning be robust against
massive label noise without any mitigation? We provide an affirmative answer
for the case of symmetric label noise: We find that certain DNNs, including
under-parameterized and over-parameterized models, can tolerate massive
symmetric label noise up to the information-theoretic threshold. By appealing
to classical statistical theory and universal consistency of DNNs, we prove
that for multiclass classification, $L_1$-consistent DNN classifiers trained
under symmetric label noise can achieve Bayes optimality asymptotically if the
label noise probability is less than $\frac{K-1}{K}$, where $K \ge 2$ is the
number of classes. Our results show that for symmetric label noise, no
mitigation is necessary for $L_1$-consistent estimators. We conjecture that for
general label noise, mitigation strategies that make use of the noisy data will
outperform those that ignore the noisy data.
