  Training convolutional neural networks (CNNs) with a strict 1-Lipschitz
constraint under the $l_{2}$ norm is useful for adversarial robustness,
interpretable gradients and stable training. 1-Lipschitz CNNs are usually
designed by enforcing each layer to have an orthogonal Jacobian matrix (for all
inputs) to prevent the gradients from vanishing during backpropagation.
However, their performance often significantly lags behind that of heuristic
methods to enforce Lipschitz constraints where the resulting CNN is not
\textit{provably} 1-Lipschitz. In this work, we reduce this gap by introducing
(a) a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last
linear layer with a 1-hidden layer MLP that significantly improves their
performance for both standard and provably robust accuracy, (b) a method to
significantly reduce the training time per epoch for Skew Orthogonal
Convolution (SOC) layers (>30\% reduction for deeper networks) and (c) a class
of pooling layers using the mathematical property that the $l_{2}$ distance of
an input to a manifold is 1-Lipschitz. Using these methods, we significantly
advance the state-of-the-art for standard and provable robust accuracies on
CIFAR-10 (gains of +1.79\% and +3.82\%) and similarly on CIFAR-100 (+3.78\% and
+4.75\%) across all networks. Code is available at
\url{https://github.com/singlasahil14/improved_l2_robustness}.
