  This paper introduces a new parameterization of deep neural networks (both
fully-connected and convolutional) with guaranteed $\ell^2$ Lipschitz bounds,
i.e. limited sensitivity to input perturbations. The Lipschitz guarantees are
equivalent to the tightest-known bounds based on certification via a
semidefinite program (SDP). We provide a ``direct'' parameterization, i.e., a
smooth mapping from $\mathbb R^N$ onto the set of weights satisfying the
SDP-based bound. Moreover, our parameterization is complete, i.e. a neural
network satisfies the SDP bound if and only if it can be represented via our
parameterization. This enables training using standard gradient methods,
without any inner approximation or computationally intensive tasks (e.g.
projections or barrier terms) for the SDP constraint. The new parameterization
can equivalently be thought of as either a new layer type (the \textit{sandwich
layer}), or a novel parameterization of standard feedforward networks with
parameter sharing between neighbouring layers. A comprehensive set of
experiments on image classification shows that sandwich layers outperform
previous approaches on both empirical and certified robust accuracy. Code is
available at \url{https://github.com/acfr/LBDN}.
