  Current SOTA adversarially robust models are mostly based on adversarial
training (AT) and differ only by some regularizers either at inner maximization
or outer minimization steps. Being repetitive in nature during the inner
maximization step, they take a huge time to train. We propose a non-iterative
method that enforces the following ideas during training. Attribution maps are
more aligned to the actual object in the image for adversarially robust models
compared to naturally trained models. Also, the allowed set of pixels to
perturb an image (that changes model decision) should be restricted to the
object pixels only, which reduces the attack strength by limiting the attack
space. Our method achieves significant performance gains with a little extra
effort (10-20%) over existing AT models and outperforms all other methods in
terms of adversarial as well as natural accuracy. We have performed extensive
experimentation with CIFAR-10, CIFAR-100, and TinyImageNet datasets and
reported results against many popular strong adversarial attacks to prove the
effectiveness of our method.
