  Data curation is commonly considered a "secret-sauce" for LLM training, with
higher quality data usually leading to better LLM performance. Given the scale
of internet-scraped corpora, data pruning has become a larger and larger focus.
Specifically, many have shown that de-duplicating data, or sub-selecting higher
quality data, can lead to efficiency or performance improvements. Generally,
three types of methods are used to filter internet-scale corpora:
embedding-based, heuristic-based, and classifier-based. In this work, we
contrast the former two in the domain of finetuning LLMs for code generation.
We find that embedding-based methods are often confounded by length, and that a
simple heuristic--pruning long files--outperforms other methods in
compute-limited regimes. Our method can yield up to a 2x efficiency benefit in
training (while matching performance) or a 3.5% absolute performance
improvement on HumanEval (while matching compute). However, we find that
perplexity on held-out long files can increase, begging the question of whether
optimizing data mixtures for common coding benchmarks (HumanEval, MBPP)
actually best serves downstream use cases. Overall, we hope our work builds
useful intuitions about code data (specifically, the low quality of extremely
long code files) provides a compelling heuristic-based method for data pruning,
and brings to light questions in how we evaluate code generation models.
