  Generally, residual connections are indispensable network components in
building CNNs and Transformers for various downstream tasks in CV and VL, which
encourages skip shortcuts between network blocks. However, the layer-by-layer
loopback residual connections may also hurt the model's robustness by allowing
unsuspecting input. In this paper, we proposed a simple yet strong backdoor
attack method - BadRes, where the residual connections play as a turnstile to
be deterministic on clean inputs while unpredictable on poisoned ones. We have
performed empirical evaluations on four datasets with ViT and BEiT models, and
the BadRes achieves 97% attack success rate while receiving zero performance
degradation on clean data. Moreover, we analyze BadRes with state-of-the-art
defense methods and reveal the fundamental weakness lying in residual
connections.
