  This work focuses on leveraging and selecting from vast, unlabeled, open data
to pre-fine-tune a pre-trained language model. The goal is to minimize the need
for costly domain-specific data for subsequent fine-tuning while achieving
desired performance levels. While many data selection algorithms have been
designed for small-scale applications, rendering them unsuitable for our
context, some emerging methods do cater to language data scales. However, they
often prioritize data that aligns with the target distribution. While this
strategy may be effective when training a model from scratch, it can yield
limited results when the model has already been pre-trained on a different
distribution. Differing from prior work, our key idea is to select data that
nudges the pre-training distribution closer to the target distribution. We show
the optimality of this approach for fine-tuning tasks under certain conditions.
We demonstrate the efficacy of our methodology across a diverse array of tasks
(NLU, NLG, zero-shot) with models up to 2.7B, showing that it consistently
surpasses other selection methods. Moreover, our proposed method is
significantly faster than existing techniques, scaling to millions of samples
within a single GPU hour. Our code is open-sourced (Code repository:
https://anonymous.4open.science/r/DV4LLM-D761/ ). While fine-tuning offers
significant potential for enhancing performance across diverse tasks, its
associated costs often limit its widespread adoption; with this work, we hope
to lay the groundwork for cost-effective fine-tuning, making its benefits more
accessible.
