  To understand what kinds of linguistic knowledge are encoded by pretrained
Chinese language models (LMs), we introduce the benchmark of Sino LINGuistics
(SLING), which consists of 38K minimal sentence pairs in Mandarin Chinese
grouped into 9 high-level linguistic phenomena. Each pair demonstrates the
acceptability contrast of a specific syntactic or semantic phenomenon (e.g.,
The keys are lost vs. The keys is lost), and an LM should assign lower
perplexity to the acceptable sentence. In contrast to the CLiMP dataset (Xiang
et al., 2021), which also contains Chinese minimal pairs and was created by
translating the vocabulary of the English BLiMP dataset, the minimal pairs in
SLING are derived primarily by applying syntactic and lexical transformations
to naturally-occurring, linguist-annotated sentences from the Chinese Treebank
9.0, thus addressing severe issues in CLiMP's data generation process. We test
18 publicly available pretrained monolingual (e.g., BERT-base-zh, CPM) and
multi-lingual (e.g., mT5, XLM) language models on SLING. Our experiments show
that the average accuracy for LMs is far below human performance (69.7% vs.
97.1%), while BERT-base-zh achieves the highest accuracy (84.8%) of all tested
LMs, even much larger ones. Additionally, we find that most LMs have a strong
gender and number (singular/plural) bias, and they perform better on local
phenomena than hierarchical ones.
