  Current large language models demonstrate deficiencies in understanding
low-resource languages, particularly the minority languages in China. This
limitation stems from the scarcity of available pre-training data. To address
this accessibility challenge, we present MC$^2$, a Multilingual Corpus of
Minority Languages in China, which is the largest open-source corpus of its
kind so far. MC$^2$ includes four underrepresented languages: Tibetan, Uyghur,
Kazakh, and Mongolian. Notably, we focus on the less common writing systems of
Kazakh and Mongolian, i.e., Kazakh Arabic script and traditional Mongolian
script, respectively, which have been long neglected in previous corpus
construction efforts. Recognizing the prevalence of language contamination
within existing corpora, we adopt a quality-centric solution for collecting
MC$^2$, prioritizing accuracy while enhancing diversity. Furthermore, we
underscore the importance of attending to the multiplicity of writing systems,
which is closely related to the cultural awareness of the resulting models. The
MC$^2$ corpus and related models are made public to the community.
