  To improve the robustness of deep classifiers against adversarial
perturbations, many approaches have been proposed, such as designing new
architectures with better robustness properties (e.g., Lipschitz-capped
networks), or modifying the training process itself (e.g., min-max
optimization, constrained learning, or regularization). These approaches,
however, might not be effective at increasing the margin in the input (feature)
space. As a result, there has been an increasing interest in developing
training procedures that can directly manipulate the decision boundary in the
input space. In this paper, we build upon recent developments in this category
by developing a robust training algorithm whose objective is to increase the
margin in the output (logit) space while regularizing the Lipschitz constant of
the model along vulnerable directions. We show that these two objectives can
directly promote larger margins in the input space. To this end, we develop a
scalable method for calculating guaranteed differentiable upper bounds on the
Lipschitz constant of neural networks accurately and efficiently. The relative
accuracy of the bounds prevents excessive regularization and allows for more
direct manipulation of the decision boundary. Furthermore, our Lipschitz
bounding algorithm exploits the monotonicity and Lipschitz continuity of the
activation layers, and the resulting bounds can be used to design new layers
with controllable bounds on their Lipschitz constant. Experiments on the MNIST,
CIFAR-10, and Tiny-ImageNet data sets verify that our proposed algorithm
obtains competitively improved results compared to the state-of-the-art.
