  The vast majority of non-English corpora are derived from automatically
filtered versions of CommonCrawl. While prior work has identified major issues
on the quality of these datasets (Kreutzer et al., 2021), it is not clear how
this impacts downstream performance. Taking representation learning in Basque
as a case study, we explore tailored crawling (manually identifying and
scraping websites with high-quality content) as an alternative to filtering
CommonCrawl. Our new corpus, called EusCrawl, is similar in size to the Basque
portion of popular multilingual corpora like CC100 and mC4, yet it has a much
higher quality according to native annotators. For instance, 66% of documents
are rated as high-quality for EusCrawl, in contrast with <33% for both mC4 and
CC100. Nevertheless, we obtain similar results on downstream NLU tasks
regardless of the corpus used for pre-training. Our work suggests that NLU
performance in low-resource languages is not primarily constrained by the
quality of the data, and other factors like corpus size and domain coverage can
play a more important role.
