  Given the prevalence of crowd sourced labor in creating Natural Language
processing datasets, these aforementioned sets have become increasingly large.
For instance, the SQUAD dataset currently sits at over 80,000 records. However,
because the English language is rather repetitive in structure, the
distribution of word frequencies in the SQUAD dataset's contexts are relatively
unchanged. By measuring each sentences distance from the co-variate distance of
frequencies of all sentences in the dataset, we identify 10,500 examples that
create a more uniform distribution for training. While fine-tuning ELECTRA [4]
on this subset of examples reaches better performance to a model trained on all
87,000 examples. Herein we introduce a methodology for systematically pruning
datasets for fine tuning reaching better out of sample performance.
