  Methods for carefully selecting or generating a small set of training data to
learn from, i.e., data pruning, coreset selection, and data distillation, have
been shown to be effective in reducing the ever-increasing cost of training
neural networks. Behind this success are rigorously designed strategies for
identifying informative training examples out of large datasets. However, these
strategies come with additional computational costs associated with subset
selection or data distillation before training begins, and furthermore, many
are shown to even under-perform random sampling in high data compression
regimes. As such, many data pruning, coreset selection, or distillation methods
may not reduce 'time-to-accuracy', which has become a critical efficiency
measure of training deep neural networks over large datasets. In this work, we
revisit a powerful yet overlooked random sampling strategy to address these
challenges and introduce an approach called Repeated Sampling of Random Subsets
(RSRS or RS2), where we randomly sample the subset of training data for each
epoch of model training. We test RS2 against thirty state-of-the-art data
pruning and data distillation methods across four datasets including ImageNet.
Our results demonstrate that RS2 significantly reduces time-to-accuracy
compared to existing techniques. For example, when training on ImageNet in the
high-compression regime (using less than 10% of the dataset each epoch), RS2
yields accuracy improvements up to 29% compared to competing pruning methods
while offering a runtime reduction of 7x. Beyond the above meta-study, we
provide a convergence analysis for RS2 and discuss its generalization
capability. The primary goal of our work is to establish RS2 as a competitive
baseline for future data selection or distillation techniques aimed at
efficient training.
