  Lipschitz constrained networks have gathered considerable attention in the
deep learning community, with usages ranging from Wasserstein distance
estimation to the training of certifiably robust classifiers. However they
remain commonly considered as less accurate, and their properties in learning
are still not fully understood. In this paper we clarify the matter: when it
comes to classification 1-Lipschitz neural networks enjoy several advantages
over their unconstrained counterpart. First, we show that these networks are as
accurate as classical ones, and can fit arbitrarily difficult boundaries. Then,
relying on a robustness metric that reflects operational needs we characterize
the most robust classifier: the WGAN discriminator. Next, we show that
1-Lipschitz neural networks generalize well under milder assumptions. Finally,
we show that hyper-parameters of the loss are crucial for controlling the
accuracy-robustness trade-off. We conclude that they exhibit appealing
properties to pave the way toward provably accurate, and provably robust neural
networks.
