  In recent years, there has been an explosive growth in multimodal learning.
Image captioning, a classical multimodal task, has demonstrated promising
applications and attracted extensive research attention. However, recent
studies have shown that image caption models are vulnerable to some security
threats such as backdoor attacks. Existing backdoor attacks against image
captioning typically pair a trigger either with a predefined sentence or a
single word as the targeted output, yet they are unrelated to the image
content, making them easily noticeable as anomalies by humans. In this paper,
we present a novel method to craft targeted backdoor attacks against image
caption models, which are designed to be stealthier than prior attacks.
Specifically, our method first learns a special trigger by leveraging universal
perturbation techniques for object detection, then places the learned trigger
in the center of some specific source object and modifies the corresponding
object name in the output caption to a predefined target name. During the
prediction phase, the caption produced by the backdoored model for input images
with the trigger can accurately convey the semantic information of the rest of
the whole image, while incorrectly recognizing the source object as the
predefined target. Extensive experiments demonstrate that our approach can
achieve a high attack success rate while having a negligible impact on model
clean performance. In addition, we show our method is stealthy in that the
produced backdoor samples are indistinguishable from clean samples in both
image and text domains, which can successfully bypass existing backdoor
defenses, highlighting the need for better defensive mechanisms against such
stealthy backdoor attacks.
