  Existing deep neural networks (DNNs) that achieve state-of-the-art (SOTA)
performance on both clean and adversarially-perturbed images rely on either
activation or weight conditioned convolution operations. However, such
conditional learning costs additional multiply-accumulate (MAC) or addition
operations, increasing inference memory and compute costs. To that end, we
present a sparse mixture once for all adversarial training (SMART), that allows
a model to train once and then in-situ trade-off between accuracy and
robustness, that too at a reduced compute and parameter overhead. In
particular, SMART develops two expert paths, for clean and adversarial images,
respectively, that are then conditionally trained via respective dedicated sets
of binary sparsity masks. Extensive evaluations on multiple image
classification datasets across different models show SMART to have up to 2.72x
fewer non-zero parameters costing proportional reduction in compute overhead,
while yielding SOTA accuracy-robustness trade-off. Additionally, we present
insightful observations in designing sparse masks to successfully condition on
both clean and perturbed images.
