  Large language models have demonstrated remarkable potential in various
tasks, however, there remains a significant scarcity of open-source models and
data for specific domains. Previous works have primarily focused on manually
specifying resources and collecting high-quality data on specific domains,
which significantly consume time and effort. To address this limitation, we
propose an efficient data collection method $\textit{Query of CC}$ based on
large language models. This method bootstraps seed information through a large
language model and retrieves related data from public corpora. It not only
collects knowledge-related data for specific domains but unearths the data with
potential reasoning procedures. Through the application of this method, we have
curated a high-quality dataset called KNOWLEDGE PILE, encompassing four major
domains, including stem and humanities sciences, among others. Experimental
results demonstrate that KNOWLEDGE PILE significantly improves the performance
of large language models in mathematical and knowledge-related reasoning
ability tests. To facilitate academic sharing, we open-source our dataset and
code, providing valuable support to the academic community.
