  Regularizing the gradient norm of the output of a neural network with respect
to its inputs is a powerful technique, rediscovered several times. This paper
presents evidence that gradient regularization can consistently improve
classification accuracy on vision tasks, using modern deep neural networks,
especially when the amount of training data is small. We introduce our
regularizers as members of a broader class of Jacobian-based regularizers. We
demonstrate empirically on real and synthetic data that the learning process
leads to gradients controlled beyond the training points, and results in
solutions that generalize well.
