  In recent years, the field of document understanding has progressed a lot. A
significant part of this progress has been possible thanks to the use of
language models pretrained on large amounts of documents. However, pretraining
corpora used in the domain of document understanding are single domain,
monolingual, or nonpublic. Our goal in this paper is to propose an efficient
pipeline for creating a big-scale, diverse, multilingual corpus of PDF files
from all over the Internet using Common Crawl, as PDF files are the most
canonical types of documents as considered in document understanding. We
analysed extensively all of the steps of the pipeline and proposed a solution
which is a trade-off between data quality and processing time. We also share a
CCpdf corpus in a form or an index of PDF files along with a script for
downloading them, which produces a collection useful for language model
pretraining. The dataset and tools published with this paper offer researchers
the opportunity to develop even better multilingual language models.
