  Efficient and effective attacks are crucial for reliable evaluation of
defenses, and also for developing robust models. Projected Gradient Descent
(PGD) attack has been demonstrated to be one of the most successful adversarial
attacks. However, the effect of the standard PGD attack can be easily weakened
by rescaling the logits, while the original decision of every input will not be
changed. To mitigate this issue, in this paper, we propose Scale-Invariant
Adversarial Attack (SI-PGD), which utilizes the angle between the features in
the penultimate layer and the weights in the softmax layer to guide the
generation of adversaries. The cosine angle matrix is used to learn angularly
discriminative representation and will not be changed with the rescaling of
logits, thus making SI-PGD attack to be stable and effective. We evaluate our
attack against multiple defenses and show improved performance when compared
with existing attacks. Further, we propose Scale-Invariant (SI) adversarial
defense mechanism based on the cosine angle matrix, which can be embedded into
the popular adversarial defenses. The experimental results show the defense
method with our SI mechanism achieves state-of-the-art performance among
multi-step and single-step defenses.
