  Jacobian and Hessian regularization aim to reduce the magnitude of the first
and second-order partial derivatives with respect to neural network inputs, and
they are predominantly used to ensure the adversarial robustness of image
classifiers. In this work, we generalize previous efforts by extending the
target matrix from zero to any matrix that admits efficient matrix-vector
products. The proposed paradigm allows us to construct novel regularization
terms that enforce symmetry or diagonality on square Jacobian and Hessian
matrices. On the other hand, the major challenge for Jacobian and Hessian
regularization has been high computational complexity. We introduce
Lanczos-based spectral norm minimization to tackle this difficulty. This
technique uses a parallelized implementation of the Lanczos algorithm and is
capable of effective and stable regularization of large Jacobian and Hessian
matrices. Theoretical justifications and empirical evidence are provided for
the proposed paradigm and technique. We carry out exploratory experiments to
validate the effectiveness of our novel regularization terms. We also conduct
comparative experiments to evaluate Lanczos-based spectral norm minimization
against prior methods. Results show that the proposed methodologies are
advantageous for a wide range of tasks.
