  Recent studies have highlighted the potential of Lipschitz-based methods for
training certifiably robust neural networks against adversarial attacks. A key
challenge, supported both theoretically and empirically, is that robustness
demands greater network capacity and more data than standard training. However,
effectively adding capacity under stringent Lipschitz constraints has proven
more difficult than it may seem, evident by the fact that state-of-the-art
approach tend more towards \emph{underfitting} than overfitting. Moreover, we
posit that a lack of careful exploration of the design space for Lipshitz-based
approaches has left potential performance gains on the table. In this work, we
provide a more comprehensive evaluation to better uncover the potential of
Lipschitz-based certification methods. Using a combination of novel techniques,
design optimizations, and synthesis of prior work, we are able to significantly
improve the state-of-the-art VRA for deterministic certification on a variety
of benchmark datasets, and over a range of perturbation sizes. Of particular
note, we discover that the addition of large ``Cholesky-orthogonalized residual
dense'' layers to the end of existing state-of-the-art Lipschitz-controlled
ResNet architectures is especially effective for increasing network capacity
and performance. Combined with filtered generative data augmentation, our final
results further the state of the art deterministic VRA by up to 8.5 percentage
points\footnote{Code is available at \url{https://github.com/hukkai/liresnet}}.
