  Large training sets have become a cornerstone of machine learning and are the
foundation for recent advances in language modeling and multimodal learning.
While data curation for pre-training is often still ad-hoc, one common paradigm
is to first collect a massive pool of data from the Web and then filter this
candidate pool down to an actual training set via various heuristics. In this
work, we study the problem of learning a data filtering network (DFN) for this
second step of filtering a large uncurated dataset. Our key finding is that the
quality of a network for filtering is distinct from its performance on
downstream tasks: for instance, a model that performs well on ImageNet can
yield worse training sets than a model with low ImageNet accuracy that is
trained on a small amount of high-quality data. Based on our insights, we
construct new data filtering networks that induce state-of-the-art image-text
datasets. Specifically, our best performing dataset DFN-5B enables us to train
state-of-the-art CLIP models for their compute budgets: among other
improvements on a variety of tasks, a ViT-H trained on our dataset achieves
84.4% zero-shot transfer accuracy on ImageNet, out-performing models trained on
other datasets such as LAION-2B, DataComp-1B, or OpenAI's WIT. In order to
facilitate further research in dataset design, we also release a new 2 billion
example dataset DFN-2B and show that high performance data filtering networks
can be trained from scratch using only publicly available data.
