  Deep networks were recently suggested to face the odds between accuracy (on
clean natural images) and robustness (on adversarially perturbed images)
(Tsipras et al., 2019). Such a dilemma is shown to be rooted in the inherently
higher sample complexity (Schmidt et al., 2018) and/or model capacity
(Nakkiran, 2019), for learning a high-accuracy and robust classifier. In view
of that, give a classification task, growing the model capacity appears to help
draw a win-win between accuracy and robustness, yet at the expense of model
size and latency, therefore posing challenges for resource-constrained
applications. Is it possible to co-design model accuracy, robustness and
efficiency to achieve their triple wins? This paper studies multi-exit networks
associated with input-adaptive efficient inference, showing their strong
promise in achieving a "sweet point" in cooptimizing model accuracy, robustness
and efficiency. Our proposed solution, dubbed Robust Dynamic Inference Networks
(RDI-Nets), allows for each input (either clean or adversarial) to adaptively
choose one of the multiple output layers (early branches or the final one) to
output its prediction. That multi-loss adaptivity adds new variations and
flexibility to adversarial attacks and defenses, on which we present a
systematical investigation. We show experimentally that by equipping existing
backbones with such robust adaptive inference, the resulting RDI-Nets can
achieve better accuracy and robustness, yet with over 30% computational
savings, compared to the defended original models.
