  Training deep networks and tuning hyperparameters on large datasets is
computationally intensive. One of the primary research directions for efficient
training is to reduce training costs by selecting well-generalizable subsets of
training data. Compared to simple adaptive random subset selection baselines,
existing intelligent subset selection approaches are not competitive due to the
time-consuming subset selection step, which involves computing model-dependent
gradients and feature embeddings and applies greedy maximization of submodular
objectives. Our key insight is that removing the reliance on downstream model
parameters enables subset selection as a pre-processing step and enables one to
train multiple models at no additional cost. In this work, we propose MILO, a
model-agnostic subset selection framework that decouples the subset selection
from model training while enabling superior model convergence and performance
by using an easy-to-hard curriculum. Our empirical results indicate that MILO
can train models $3\times - 10 \times$ faster and tune hyperparameters
$20\times - 75 \times$ faster than full-dataset training or tuning without
compromising performance.
