  Label noise poses a serious threat to deep neural networks (DNNs). Employing
robust loss functions which reconcile fitting ability with robustness is a
simple but effective strategy to handle this problem. However, the widely-used
static trade-off between these two factors contradicts the dynamics of DNNs
learning with label noise, leading to inferior performance. Therefore, we
propose a dynamics-aware loss (DAL) to solve this problem. Considering that
DNNs tend to first learn beneficial patterns, then gradually overfit harmful
label noise, DAL strengthens the fitting ability initially, then gradually
improves robustness. Moreover, at the later stage, to further reduce the
negative impact of label noise and combat underfitting simultaneously, we let
DNNs put more emphasis on easy examples than hard ones and introduce a
bootstrapping term. Both the detailed theoretical analyses and extensive
experimental results demonstrate the superiority of our method. Our source code
can be found in https://github.com/XiuchuanLi/DAL.
