  Compute-efficient training of large language models (LLMs) has become an
important research problem. In this work, we consider data pruning as a method
of data-efficient training of LLMs, where we take a data compression view on
data pruning. We argue that the amount of information of a sample, or the
achievable compression on its description length, represents its sample
importance. The key idea is that, less informative samples are likely to
contain redundant information, and thus should be pruned first. We leverage
log-likelihood function of trained models as a surrogate to measure information
content of samples. Experiments reveal a surprising insight that
information-based pruning can enhance the generalization capability of the
model, improves upon language modeling and downstream tasks as compared to the
model trained on the entire dataset.
