  Pre-trained Natural Language Processing (NLP) models can be easily adapted to
a variety of downstream language tasks. This significantly accelerates the
development of language models. However, NLP models have been shown to be
vulnerable to backdoor attacks, where a pre-defined trigger word in the input
text causes model misprediction. Previous NLP backdoor attacks mainly focus on
some specific tasks. This makes those attacks less general and applicable to
other kinds of NLP models and tasks. In this work, we propose \Name, the first
task-agnostic backdoor attack against the pre-trained NLP models. The key
feature of our attack is that the adversary does not need prior information
about the downstream tasks when implanting the backdoor to the pre-trained
model. When this malicious model is released, any downstream models transferred
from it will also inherit the backdoor, even after the extensive transfer
learning process. We further design a simple yet effective strategy to bypass a
state-of-the-art defense. Experimental results indicate that our approach can
compromise a wide range of downstream NLP tasks in an effective and stealthy
way.
