  Recent studies show that pre-trained language models (LMs) are vulnerable to
textual adversarial attacks. However, existing attack methods either suffer
from low attack success rates or fail to search efficiently in the
exponentially large perturbation space. We propose an efficient and effective
framework SemAttack to generate natural adversarial text by constructing
different semantic perturbation functions. In particular, SemAttack optimizes
the generated perturbations constrained on generic semantic spaces, including
typo space, knowledge space (e.g., WordNet), contextualized semantic space
(e.g., the embedding space of BERT clusterings), or the combination of these
spaces. Thus, the generated adversarial texts are more semantically close to
the original inputs. Extensive experiments reveal that state-of-the-art (SOTA)
large-scale LMs (e.g., DeBERTa-v2) and defense strategies (e.g., FreeLB) are
still vulnerable to SemAttack. We further demonstrate that SemAttack is general
and able to generate natural adversarial texts for different languages (e.g.,
English and Chinese) with high attack success rates. Human evaluations also
confirm that our generated adversarial texts are natural and barely affect
human performance. Our code is publicly available at
https://github.com/AI-secure/SemAttack.
