  In this paper, we focus on studying robustness evaluation of Chinese question
matching. Most of the previous work on analyzing robustness issue focus on just
one or a few types of artificial adversarial examples. Instead, we argue that
it is necessary to formulate a comprehensive evaluation about the linguistic
capabilities of models on natural texts. For this purpose, we create a Chinese
dataset namely DuQM which contains natural questions with linguistic
perturbations to evaluate the robustness of question matching models. DuQM
contains 3 categories and 13 subcategories with 32 linguistic perturbations.
The extensive experiments demonstrate that DuQM has a better ability to
distinguish different models. Importantly, the detailed breakdown of evaluation
by linguistic phenomenon in DuQM helps us easily diagnose the strength and
weakness of different models. Additionally, our experiment results show that
the effect of artificial adversarial examples does not work on the natural
texts.
