  Pre-trained vision models (PVMs) have become a dominant component due to
their exceptional performance when fine-tuned for downstream tasks. However,
the presence of backdoors within PVMs poses significant threats. Unfortunately,
existing studies primarily focus on backdooring PVMs for the classification
task, neglecting potential inherited backdoors in downstream tasks such as
detection and segmentation. In this paper, we propose the Pre-trained Trojan
attack, which embeds backdoors into a PVM, enabling attacks across various
downstream vision tasks. We highlight the challenges posed by cross-task
activation and shortcut connections in successful backdoor attacks. To achieve
effective trigger activation in diverse tasks, we stylize the backdoor trigger
patterns with class-specific textures, enhancing the recognition of
task-irrelevant low-level features associated with the target class in the
trigger pattern. Moreover, we address the issue of shortcut connections by
introducing a context-free learning pipeline for poison training. In this
approach, triggers without contextual backgrounds are directly utilized as
training data, diverging from the conventional use of clean images.
Consequently, we establish a direct shortcut from the trigger to the target
class, mitigating the shortcut connection issue. We conducted extensive
experiments to thoroughly validate the effectiveness of our attacks on
downstream detection and segmentation tasks. Additionally, we showcase the
potential of our approach in more practical scenarios, including large vision
models and 3D object detection in autonomous driving. This paper aims to raise
awareness of the potential threats associated with applying PVMs in practical
scenarios. Our codes will be available upon paper publication.
