  Backdoor attacks have been shown to impose severe threats to real
security-critical scenarios. Although previous works can achieve high attack
success rates, they either require access to victim models which may
significantly reduce their threats in practice, or perform visually noticeable
in stealthiness. Besides, there is still room to improve the attack success
rates in the scenario that different poisoned samples may have different target
labels (a.k.a., the all-to-all setting). In this study, we propose a novel
imperceptible backdoor attack framework, named Impart, in the scenario where
the attacker has no access to the victim model. Specifically, in order to
enhance the attack capability of the all-to-all setting, we first propose a
label-specific attack. Different from previous works which try to find an
imperceptible pattern and add it to the source image as the poisoned image, we
then propose to generate perturbations that align with the target label in the
image feature by a surrogate model. In this way, the generated poisoned images
are attached with knowledge about the target class, which significantly
enhances the attack capability.
