  Almost all the state-of-the-art neural networks for computer vision tasks are
trained by (1) pre-training on a large-scale dataset and (2) finetuning on the
target dataset. This strategy helps reduce dependence on the target dataset and
improves convergence rate and generalization on the target task. Although
pre-training on large-scale datasets is very useful, its foremost disadvantage
is high training cost. To address this, we propose efficient filtering methods
to select relevant subsets from the pre-training dataset. Additionally, we
discover that lowering image resolutions in the pre-training step offers a
great trade-off between cost and performance. We validate our techniques by
pre-training on ImageNet in both the unsupervised and supervised settings and
finetuning on a diverse collection of target datasets and tasks. Our proposed
methods drastically reduce pre-training cost and provide strong performance
boosts. Finally, we improve standard ImageNet pre-training by 1-3% by tuning
available models on our subsets and pre-training on a dataset filtered from a
larger scale dataset.
