  Data curation is an essential component of large-scale pretraining. In this
work, we demonstrate that jointly selecting batches of data is more effective
for learning than selecting examples independently. Multimodal contrastive
objectives expose the dependencies between data and thus naturally yield
criteria for measuring the joint learnability of a batch. We derive a simple
and tractable algorithm for selecting such batches, which significantly
accelerate training beyond individually-prioritized data points. As performance
improves by selecting from larger super-batches, we also leverage recent
advances in model approximation to reduce the associated computational
overhead. As a result, our approach--multimodal contrastive learning with joint
example selection (JEST)--surpasses state-of-the-art models with up to
13$\times$ fewer iterations and 10$\times$ less computation. Essential to the
performance of JEST is the ability to steer the data selection process towards
the distribution of smaller, well-curated datasets via pretrained reference
models, exposing the level of data curation as a new dimension for neural
scaling laws.
