  Researchers investigating example hardness have increasingly focused on the
dynamics by which neural networks learn and forget examples throughout
training. Popular metrics derived from these dynamics include (i) the epoch at
which examples are first correctly classified; (ii) the number of times their
predictions flip during training; and (iii) whether their prediction flips if
they are held out. However, these metrics do not distinguish among examples
that are hard for distinct reasons, such as membership in a rare subpopulation,
being mislabeled, or belonging to a complex subpopulation. In this paper, we
propose $second$-$split$ $forgetting$ $time$ (SSFT), a complementary metric
that tracks the epoch (if any) after which an original training example is
forgotten as the network is fine-tuned on a randomly held out partition of the
data. Across multiple benchmark datasets and modalities, we demonstrate that
$mislabeled$ examples are forgotten quickly, and seemingly $rare$ examples are
forgotten comparatively slowly. By contrast, metrics only considering the first
split learning dynamics struggle to differentiate the two. At large learning
rates, SSFT tends to be robust across architectures, optimizers, and random
seeds. From a practical standpoint, the SSFT can (i) help to identify
mislabeled samples, the removal of which improves generalization; and (ii)
provide insights about failure modes. Through theoretical analysis addressing
overparameterized linear models, we provide insights into how the observed
phenomena may arise. Code for reproducing our experiments can be found here:
https://github.com/pratyushmaini/ssft
