  We investigate security concerns of the emergent instruction tuning paradigm,
that models are trained on crowdsourced datasets with task instructions to
achieve superior performance. Our studies demonstrate that an attacker can
inject backdoors by issuing very few malicious instructions (~1000 tokens) and
control model behavior through data poisoning, without even the need to modify
data instances or labels themselves. Through such instruction attacks, the
attacker can achieve over 90% attack success rate across four commonly used NLP
datasets. As an empirical study on instruction attacks, we systematically
evaluated unique perspectives of instruction attacks, such as poison transfer
where poisoned models can transfer to 15 diverse generative datasets in a
zero-shot manner; instruction transfer where attackers can directly apply
poisoned instruction on many other datasets; and poison resistance to continual
finetuning. Lastly, we show that RLHF and clean demonstrations might mitigate
such backdoors to some degree. These findings highlight the need for more
robust defenses against poisoning attacks in instruction-tuning models and
underscore the importance of ensuring data quality in instruction
crowdsourcing.
