  Revealing the robustness issues of natural language processing models and
improving their robustness is important to their performance under difficult
situations. In this paper, we study the robustness of paraphrase identification
models from a new perspective -- via modification with shared words, and we
show that the models have significant robustness issues when facing such
modifications. To modify an example consisting of a sentence pair, we either
replace some words shared by both sentences or introduce new shared words. We
aim to construct a valid new example such that a target model makes a wrong
prediction. To find a modification solution, we use beam search constrained by
heuristic rules, and we leverage a BERT masked language model for generating
substitution words compatible with the context. Experiments show that the
performance of the target models has a dramatic drop on the modified examples,
thereby revealing the robustness issue. We also show that adversarial training
can mitigate this issue.
