  We report a series of robust empirical observations, demonstrating that deep
Neural Networks learn the examples in both the training and test sets in a
similar order. This phenomenon is observed in all the commonly used benchmarks
we evaluated, including many image classification benchmarks, and one text
classification benchmark. While this phenomenon is strongest for models of the
same architecture, it also crosses architectural boundaries -- models of
different architectures start by learning the same examples, after which the
more powerful model may continue to learn additional examples. We further show
that this pattern of results reflects the interplay between the way neural
networks learn benchmark datasets. Thus, when fixing the architecture, we show
synthetic datasets where this pattern ceases to exist. When fixing the dataset,
we show that other learning paradigms may learn the data in a different order.
We hypothesize that our results reflect how neural networks discover structure
in natural datasets.
